Goodfire Research 论文解释为何更大模型能学到小模型学不会的任务

Abstract

Larger models learn tasks smaller models do not. What drives this phenomenon? We develop a simple phenomenological argument that power-law scaling already suggests that a larger model will be able to learn a part of the data distribution that a smaller model fails to learn, even with infinite training data. To validate this claim and identify its causes, we study the effects of model scaling on a synthetic setup consisting of a mixture of tasks that show monotonic scaling curves. The results point to a data-induced competition over resources (neurons). Specifically, smaller models allocate their neurons to high frequency or low complexity tasks, and so they learn solutions that perform poorly on rare and complex tasks. Moreover, this happens even when solutions capable of expressing the desired task exist. We then assess how a larger model circumvents this data-centric bottleneck, finding that it traces to a reduced interference mechanism: larger models can allocate enough resources to common tasks that the gradient updates for those tasks become weak, which means that they do not overwrite rare-task features as they slowly accumulate. Finally, to further validate these claims, we pretrain OLMo models (4M to 4B parameters) on novel tasks of varying frequency and complexity. The results mirror those from our synthetic data experiments: only the larger OLMo models learn the infrequent and complex tasks, and these larger models embed more task features in their representations and show less gradient interference between tasks. Overall, we offer a data-centric account of why larger models learn tasks that smaller models fail to. This helps explain why larger models are better in practice, and it can inform practical questions concerning model sizing and training data mixtures.

1 Introduction

Modern machine learning is celebrated for its massive generalist models, which are capable of handling arbitrary inputs in diverse and complex environments [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]. Based on the empirical finding that larger models often excel where smaller¹¹ 1 We use the terms “larger” and “smaller” informally here but develop a precise relational definition of these terms in Sec. 2. models show random-chance performance, prior work has claimed that the ability to solve certain critical tasks only emerges in larger models [11, 12, 13, 14, 15, 16, 17, 18, 19]. Such arguments have fueled the drive towards increased scaling. However, given the large training and inference costs that large models impose, it is worth identifying precisely what marginal benefits are unlocked by larger models and whether scaling parameters is the sole way of realizing those benefits.

Our argument begins from the observation that power-law scaling [20, 21, 22] already suggests that there is a regime in which a smaller model fails to learn parts of a data mixture that a larger model succeeds on, even under asymptotic training (Fig. 1, Sec. 2). This suggests that larger models enjoy a genuine advantage that may allow them to learn task distributions that smaller models will inevitably fail to learn within the same training setup. Importantly, this is not an argument that larger models are simply more sample efficient [23, 24, 25, 26, 18, 27, 28, 29], but rather that smaller models suffer from a more fundamental limitation even under infinite training regimes.

To validate this prediction and identify its causes, we analyze a setting involving a mixture of regression tasks. In this, we are inspired by much recent work using toy tasks to pinpoint the effects of scaling [30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40]. Furthermore, all of the individual tasks in our setting are learnable by the models under consideration, capturing the idea that tasks smaller models fail to learn can still be instilled into them via post-training [41, 42, 43, 44, 45, 46, 47, 48, 49, 50]. Correspondingly, mere expressivity notions are not the issue; instead, the question concerns the ability of these models to learn complex task distributions from data. These experiments lead to two key findings, as described below.

First, scaling enables learning rare and complex tasks (Sec. 3.1). Our experimental setting defines controlled manipulations of task frequency and complexity. We present an analytic argument that only larger models will (on average) learn the rare and complex tasks present in this setting, and we verify this analysis experimentally (Fig. 2).

Second, reduced competition for resources enables learning rare and complex tasks (Sec. 3.2). Here, we extend our formal analysis to show that, upon observation of samples from a rare task, model parameters update, but only larger models, by virtue of having more parameters and hence less gradient interference, are able to retain memory of a previously observed batch of data from a rare task. Thus, when the next batch of rare-task data comes in, the larger model builds on its prior knowledge, which ultimately leads to success despite the impoverished learning signal. In contrast, the smaller model is forced to start from scratch and consequently fails. We again verify these findings experimentally in our regression setting (Figs. 3 and 4).

Finally, we validate the above theoretical arguments in real LLMs (Sec. 4). Specifically, we pretrain OLMo models (4M to 4B parameters) on the Dolma v1.7 corpus with completely novel tasks injected at controlled frequency. We find that only the larger OLMo models are able to learn the infrequent and complex tasks (Sec. 4.2). Furthermore, these OLMo models mirror our toy-task models in deeper ways: larger OLMo models have more task features embedded in their representations (Sec. 4.3) and show less gradient interference (Sec. 4.4). Beyond supporting our theoretical claims, these results can provide practical guidance to large-scale model training efforts.

Overall, the data-centric nature of our analysis suggests that understanding why larger models learn more requires not only asking what they can represent, but also what is learnable under gradient-based optimization from a given data mixture.

2 A Phenomenological Model Predicts Larger Models Learn More

Neural network scaling is known to predictably and monotonically improve loss [28, 20, 51]:

where L0 denotes the irreducible loss, A,B are constants, and α,β are parameter / data exponents (α≈0.46 and β≈0.51 for Chinchilla-scaling [28]). Training in a compute-optimal manner, i.e., finding the model size and data configuration that helps achieve the minimum loss at a given compute budget C, gives us

where γ=0.34, and LC​(N) denotes the optimum loss achieved when training a model with N parameters under resource constraints. The relation shows larger models are expected to achieve a smaller loss. However, resource-constrained training by itself does not inform what a model can actually express. Specifically, even though a smaller model may have a worse compute-optimal loss, we do not know if it is fundamentally incapable of achieving the same loss as the larger model. To assess that statement, we must evaluate a model’s loss under asymptotic resources (i.e., infinite data):²² 2 We note power-law scaling need not hold asymptotically [52, 31], which is why we call this argument phenomenological. It motivates the subsequent, rigorous claims.

If α>γ, as is the case in practice, we again see gains from merely scaling the model size. That is, the asymptotic loss achieved by a larger model is better than the smaller one. This indicates there is a part of the training distribution a smaller model, despite observing infinite data, fails to learn. Based on this phenomenological argument, we define the following.

Definition 1( Learnable via data scaling).

Consider a target model with Nl number of parameters that we call “large”. We say a “smaller” model, i.e., for which parameter count Ns<Nl, can recover the loss of a larger model via data scaling if LC​(Ns)−LC​(Nl)>0, but L∞​(Ns)−LC​(Nl)<0.

Def. 1 thus captures the scenario put forward in Sec. 1. That is, the smaller model may in fact be just undertrained: the larger model learns more sample efficiently and reduces loss faster, but a smaller model can eventually catch up [23, 24, 25, 26, 18, 27, 28, 29]. Correspondingly, the marginal ability of a larger model to explain the data distribution (i.e., the loss) can be recovered by a smaller model merely observing more data. Nevertheless, there exist regimes where data scaling will not suffice, as described next.

Definition 2( Learnable via model scaling).

Consider a target model with Nl number of parameters that we call “large”. For a small scalar value ϵ, we define Ns∗​(ϵ) as the largest “small” model if L∞​(Ns∗​(ϵ))−LC​(Nl)>ϵ. That is, even asymptotically, the smallest model never reaches the same loss as the large model. Correspondingly, for a given model size N, we call it “small” if N<Ns∗​(ϵ) and say recovering the loss of the larger model requires model scaling.

This latter scenario thus captures the case where, when two models with parameter counts Ns,Nl, with Ns<Nl, are trained, there is truly a marginal improvement for explaining the data that can be attributed to the larger model having more parameters. This is the most interesting case that warrants further study: what is it about the data that only a larger model can learn, such that the smaller model cannot, even after observing infinite data? How precisely does having more parameters aid this learning? We aim to answer these questions in the following sections.

3 Scaling Allows Learning Rare Tasks by Reducing Gradient Interference

Our phenomenological argument in Sec. 2 motivates the claim that larger models are likely to learn a part of the data distribution smaller models will fail to learn. We next aim to get more concrete about this claim. Specifically, we exploit the fact that our argument is merely based on monotonic (power-law) scaling—a phenomenon even synthetic tasks can recapitulate [30, 31, 32, 33, 34, 35, 37, 38, 39, 40]. Such tasks have in fact been used in prior work to make accurate predictions about scaling behavior for large-scale models [51, 52]. We thus follow this line of work and develop a multi-task learning setup that helps assess which tasks a larger model can learn but a smaller model cannot. We generalize our claims to an off-the-shelf language model pretraining pipeline [53] in Sec. 4, finding the core hypotheses derived out of this toy setting hold true on even a large-scale training pipeline.

Data.

We consider a multi-task learning setup where samples are drawn from a mixture of K linear regression tasks. Specifically, the kth task is assumed to appear with frequency πk>0, such that ∑kπk=1, and has covariance Ck=Bk​Λk​Bk⊤=∑j≥1λk,j​bk,j​bk,j⊤. Here, the “feature matrix” Bk=[bk,1,bk,2,…] is assumed to have orthonormal columns; Λk=diag⁡(λk,1,λk,2,…) with λk,1≥λk,2≥⋯≥0; and different tasks occupy orthogonal blocks, i.e., Bk⊤​Bℓ=0 for k≠ℓ. If the spectrum {λk,j} decays slowly, the task requires more directions for producing the corresponding target—we can thus compare the relative complexity of two tasks by comparing the rate at which their spectra decay. Compared to prior work studying theory of scaling laws based on toy regression tasks, we emphasize that our setup involves the learning of multiple tasks simultaneously.

Teacher / Student Models.

For a given input x∼𝒩⁡(0,I), the teacher for task k is defined as yk=Λk1/2​Bk⊤​x. The student uses a shared width-N encoder U∈ℝd×N, U⊤​U=I, with projector PU=U​U⊤, together with task-specific linear decoders Dk to discern between tasks. Correspondingly, the student prediction is y^k=Dk​U⊤​x. The total mixture loss is the weighted sum ℒN​(U)=∑k=1Kπk​ℓk​(U), where ℓk​(U)=𝔼⁡[‖yk−Dk​U⊤​x‖22] is loss of the kth task. Note that herein, since the optimal decoder admits a closed-form solution Dk∗=Λk1/2​Bk⊤​U, we solely analyze the dynamics of the encoder, which produces features used by the student for making predictions.

3.1 Larger Models Learn Rarer, More Complex Tasks

In order to narrow down a mechanism that explains how larger models may be able to learn more, we must first identify precisely what it is that a larger model learns but a smaller one fails to. We begin with answering this question in our toy setup.

Theorem 3(Features are Learned in Order of Utility).

For a given U, the mixture loss reduces to LN​(U)=Tr⁡(M)−Tr⁡(U⊤​M​U), where M:=∑k=1Kπk​Ck. Hence, a width-N minimizer spans the top-N eigenspace of M, whose eigenvalues are defined by the weighted per-task spectra:

Thus, the optimal encoder keeps the N features (k,j) with largest uk,j—we call these terms utilities. This implies if nk​(N) denotes the number of retained features from task k, then ℓk∗​(N)=∑j>nk​(N)λk,j. Conversely, the minimum width at which a model learns at least m features for all tasks is N∗​(m)=min⁡{N:nk​(N)≥m}.

In the context of our toy task, the statement above helps answer the question ‘‘what does width buy?’’ by defining a concrete ranking rule for feature learning.³³ 3 This claim can also be seen as a static ordering rule that local optima visited by a model during training will be expected to dynamically follow in its saddle-to-saddle dynamics [54, 55, 56, 57] Specifically, it says a larger model, asymptotically, learns exactly those features whose utilities are lower than those of the features learned by a smaller model. This implies if a task is observed infrequently or it involves several features, e.g., if its spectrum decays very slowly, then (on average) only a larger model will learn it.

Verification.

We verify the claim above by training our student model on a mixture of K=32 tasks, using the Adam optimizer for 100K steps (the loss does not improve beyond this budget even when trained up to 10× longer; see Fig. 21). We use a power-law prior k−β to define task frequencies, and a power-law per-task spectrum λk,j∝j−α. For simplicity of visualization, we let α=2 be shared across tasks and only vary task frequencies by changing β (see App. D for experiments modulating complexity by varying α). Results are reported in Fig. 2 (also see App. E for further results). We find (a) the per-task loss and (b) the overall residual loss predictably reduce with model width. Critically, we see larger models learn infrequent tasks better than smaller ones.

3.2 Scaling Reduces Interference and Allows for Retention of Rare Task Observations

While the argument above—i.e., a larger model learns low utility, infrequent features—is intuitively reasonable, it is critical to note that if the frequency at which a task or its features are seen is very low, then, regardless of size, there is a statistical bottleneck here that a model needs to circumvent. For example, in the experiments shown in Fig. 2b, a model must learn a task that constitutes merely 0.25% of observations. We next analyze how width helps surmount this challenge. To this end, note that for the kth task, the Riemannian gradient is Gk​(U)=2​(I−PU)​Ck​U, and hence the mixture gradient is U˙=2​(I−PU)​M​U. We then have the following claim.

Theorem 4(Residual Controls Learning).

Let 𝖥⊆[K] denote the common or frequent tasks. Define these tasks’ weighted covariance M𝖥:=∑k∈𝖥πk​Ck and residual signal δ𝖥​(U):=Tr⁡((I−PU)​M𝖥). Then, the aggregate common-task gradient G𝖥​(U)=2​(I−PU)​M𝖥​U obeys the bound

The statement above says a set of tasks move the model only through the part of their covariance that is not already explained by the current representation, i.e., the residual δ𝖥​(U). Correspondingly, once the high-utility common-task features have been learned, their updates become weak (i.e., low norm). This leaves any spare width available to rare tasks. More precisely, let μ1𝖥≥μ2𝖥≥⋯ be the eigenvalues of M𝖥. The best width-N representation for the common tasks alone leaves residual δ𝖥∗​(N)=∑i>Nμi𝖥. Then, via Theorem 4, we get the following.

Corollary 5(Width-Scaling Reduces Competition).

Define N𝖥​(ε):=min⁡{N:∑i>Nμi𝖥≤ε}. For every N≥N𝖥​(ε), there exists an encoder for which δ𝖥∗​(N)≤ϵ and ‖G𝖥​(U)‖F≤2​μ1𝖥​ϵ.

That is, once N≳N𝖥​(ε), the model contains enough resources that can be allocated to the common tasks, rendering the gradient towards them weak. This makes the remaining resources available to rare tasks. However, even once interference is weak enough for a rare task to be learned, it is unclear whether gradient descent can actually consolidate that signal across its infrequent observations. To this end, we next characterize the local condition under which a specific rare feature can pull the model towards itself, without forcing the forgetting of well-learned tasks. Specifically, assume we wanted to learn a rare rank-one task Cr=λr​br​br⊤ orthogonal to the common block. Let UF(N) be top-N eigenspace of MF with eigenvalues μ1F≥μ2F≥⋯. Then, we have the following claim.

Proposition 6(Interference Reduces via Scaling).

The common-task solution U𝖥(N) is stable against direction br, i.e., common tasks’ loss does not grow by learning of br, iff πr​λr<μN𝖥. Thus, the critical width at which br gets learned is Nrcrit:=min⁡{N:μN𝖥≤πr​λr}.

The claim above hence shows width scaling helps in two related but distinct ways. First, it reduces the total unresolved common-task signal δ𝖥∗​(N), which bounds the aggregate common-task gradient. Second, as Proposition 6 shows, it lowers the weakest occupied common-task utility μN𝖥, which determines whether a particular rare feature can displace a common feature and become locally stable; if the rare feature’s utility is lower, even if the model updates to learn it, the common tasks’ least utility feature will eventually replace it. This will result in a swinging, update-and-forget learning dynamic where the rare task features and lowest utility features of common tasks will compete over model parameters. Overall, this suggests the learning bottleneck is defined by the interaction between data and scale: if the task we care to learn does not have sufficient utility for reducing the loss, then the model will prefer to learn and preserve lower-order modes of other tasks; however, by increasing width, one avails capacity to such low-utility tasks and reduces competition between tasks over model parameters, enabling learning of the rare task without forcing the forgetting of features relevant to common tasks.

Validation.

We train models of varying width on the same setup as Fig. 2 and plot how much signal from directions describing a task is present in the model’s intermediate representation. Specifically, since ℓk​(U)=Tr⁡((I−PU)​Ck), the signal captured for task k is Tr⁡(PU​Ck), where Tr(.) denotes trace of a matrix. We thus measure sk​(U)=Tr⁡(PU​Ck)Tr⁡(Ck). To contextualize this value, we normalize with respect to a random baseline, yielding s~k​(U)=sk​(U)−N/d1−N/d; N/d denotes the expected value of sk​(U) if U were a randomly drawn matrix from the Stiefel manifold. We call this quantity “Normalized Signal”. Results are shown in Fig. 3. We see when the model width is small, frequent tasks have a high residual signal remaining to be explained; here the set of frequent tasks is defined as top-K tasks whose prior sums to 0.8, resulting in K=3. Correspondingly, rare tasks’ signal in model representation is no better than random. Meanwhile, as we scale, once the width crosses our predicted threshold δF∗​(Nrcrit), we find the bulk of the frequent tasks’ signal is explained away and rare tasks start to get learned.

To isolate how the gap between observations interacts with width, we also design a matched-frequency injection experiment: the rare task is excluded from training for G steps, then injected in a batch enlarged to m=G⋅B⋅ρr rare samples so that its long-run frequency exactly matches the setup of Fig. 2. This emphasizes the ability of a model to retain memories about observed data, while preserving the total frequency with which it is seen. Results are shown in Fig. 4. We see at the end of training, rare-task signal decays monotonically with G at all widths, but far more steeply for smaller models. Meanwhile, the learning dynamics in panel (b) show that after each injection, a larger model accumulates rare-task signal and retains enough of it to build on the next injection, while a smaller model decays back to near-zero in between (an intuitive model explaining this dynamic is shown in Fig. 11 and analytically described in App. C.4). Overall, our results showing how larger models learn tasks smaller models do not can be summarized as follows.

4 Corroborating Claims with the OLMo Pretraining Pipeline

We now verify the claims of Sec. 3 in a realistic LLM pre-training setting using the OLMo pipeline. We train models of size 4M to 4B on up to 210B tokens (∼50K steps). Following the structure of Sec. 3, we offer analyses at three levels: loss, representation, and gradient.

4.1 Setup

A key variable in our claims is the frequency of a task.⁴⁴ 4 Defining the complexity of a natural task is difficult, and hence we solely focus on frequency in this section. However, measuring the frequency of a naturally occurring task in pre-training data is challenging, as instances from the same task can occur in many surface forms. To tightly control task frequency, we adopt a data injection framework from the memorization literature [58, 59, 60, 61]. We inject different instances sampled from the distribution of a “special” task T at a controlled frequency f to measure whether a model has learned the task distribution. The task T is special in the sense that it is unlikely to be part of normal pre-training data. We then train models of various sizes on data mixtures generated from different values of f.

Tasks.

We consider two special tasks T: comparison (TCMP) and modular addition (TADD). Both tasks are encoded as a sequence of three tokens: TOK1, TOK2, LABEL, where TOK1, TOK2 ∈𝒮, a set of 100 tokens randomly sampled from the vocab. There are exactly 10K instances per task, which are split 50/50 for training and testing. Critically, both tasks require models to learn certain geometrical structures to generalize [62]. This provides a measure for learning a task (as opposed to memorizing training instances) and a set of features to verify the interference hypothesis of Sec. 3.

Data.

We use Dolma v1.7 as the pre-training corpus [63]. Given a task T, we inject instances sampled from its train split at a frequency of 7.8×10−3 to 2.4×10−8, roughly from 1K instances per batch to 1 instance every 10 batches. To ensure the injected task frequency is comparable to the frequency of tasks learned in pre-training, we sample two reference tasks Rcmp and Radd from pre-training that involve similar high-level functions. The three-token sequence plus an end of document token replace the first four tokens of a training sequence. See App. B.3 for further details on the experimental setup.

Models.

We train OLMo models [64] with 4M, 20M, 300M, 1B, and 4B parameters. We focus on scaling the models’ hidden and MLP dimensions and the number of attention heads; the 4M parameter model has depth 8 and the rest have depth 16. See App. B.2 for further details.

4.2 Behavioral Evidence

Larger Models Learn Rarer Tasks.

We first replicate the behavioral findings in Sec. 3.1. We measure the effect of task frequency by comparing multiple training runs that only differ by the frequency of the injected task. As shown in Fig. 5, larger models learn lower-frequency tasks much better than smaller models do. This matches the pattern in Fig. 2. Moreover, tasks are learned in the order of frequency. For each model run, we compare the order in which the injected task TCMP and the reference tasks are learned, as shown in Fig. 6a. Most importantly, larger models do not just lead to better memorization of training instances, i.e., low training loss, but also learn generalizable task structures, i.e., high eval accuracy. On TADD, only larger models trained on higher frequency exhibit the grokking phenomenon [65].

Rare-Task Retention Has an Effect on Learning.

We conduct the matched-frequency injection experiment as described in Fig. 4, i.e., injecting N task instances every N batches, for N=1,10,20,50,100. Fig. 6b shows the effects of retention on learning, as models trained with larger gaps between task instances have higher task loss, even though the global task frequency of all runs is equivalent.

4.3 Representational Evidence

Task Features.

In our toy setting (Sec. 3), we know analytically which features are necessary for learning the kth task, i.e., Bk, and to what extent the model can represent these features, i.e., PU. For our OLMo models, we can empirically identify a set of causal features that a pre-trained LM would use to solve the task and localize them in the model representations. Specifically, for TCMP, the task feature of core relevance is the global order of the tokens, which allows number comparisons; meanwhile for TADD, task features are the Fourier modes [66, 67, 68], as shown in Fig. 7. We visualize the feature geometry using the largest model trained on the most frequent task data. These task features allow us to conduct versions of the gradient and representation-level analyses in Sec. 3.

More Task Features are Present in Larger Model Representations.

We first localize the task features in models that have clearly learned the task. We then measure to what extent these target task features are present in all models, which parallels the metric ℓk​(U) used in the toy setting. For localization, we use distributed alignment search (DAS) [69] which finds subspaces that causally encode the features. For TCMP, a global ordering of the tokens can be localized to a 1-D subspace in the residual stream of the first layer. For TADD, Fourier modes can be identified in the residual stream from earlier layers to the last layer. We then use task-specific metrics to measure to what extent these task features are present in model representations. For TCMP, since the geometry of the task feature is a single direction, we apply linear regression to representations spanning the top K=50 principal components. For TADD, we measure the total number of Fourier modes present through all layers. We include the details in App. B.4. The last two columns of Fig. 7 show the extent to which the target task features are present in each model across checkpoints. For the frequency (middle) column, we fix the model size to 1B. For the model size (last) column, we fix the task frequency to be 2.4×10−6 for TCMP and 7.8×10−3 for TADD. We see that (i) the presence of task features is highly correlated with high accuracy on the test set, and (ii) larger models and models trained on more frequent task data clearly learn these task features faster.

4.4 Gradient Evidence

We now connect the behavioral evidence (Sec. 4.2) and the internal representation account (Sec. 4.3) by analyzing how task gradients interfere with non-task gradients on a set of neurons that implement the task circuit. We focus on TCMP training runs in Fig. 8, where 100 task instances are injected every 100 steps.

Task Neurons.

We first identify which MLP layers implement the task features defined in Sec. 4.3. For all the models that we compared, the first layer MLP has the largest causal effects on task predictions. We further identify the top K neurons in the first layer MLP that have the largest gradient magnitude and use the gradients of these neurons for analysis. Details can be found in App. B.5.

Task Reference Direction gr.

We estimate the task reference direction using the aggregated gradient of the task loss computed over all 10K task instances, an analogy to Gr in the toy setting. This direction may shift across training steps; however, at a given step, it is the optimal task direction.

Larger Models Show Less Gradient Interference Between General Language Modeling and Our Injected Task.

We quantify the relation between the task reference gr and the batch gradient g, which can be further decomposed into gradient from the task tokens gt (if exists in batch) and non-task tokens g𝑛𝑡, i.e., g=gt+g𝑛𝑡. We first measure the cosine similarity between task reference and batch gradient direction, replicating the results in Fig. 4. We additionally analyze whether task or non-task tokens contribute to this similarity; while task token gradient aligning with task reference gr is expected, non-task token gradient with non-zero cosine similarity suggests that the language modeling direction is interfering with the task gradient direction.

Results are shown in Fig. 9. In the top panel, larger models have higher similarity between g and gr at the injection steps, 0.08±0.02 for the 1B model and 0.04±0.04 for the 300M model, the similarity typically regresses towards zero between injections. For the 20M model, the similarity scores oscillate wildly across batches, even at the injection step. In fact, the high similarity between non-task gradient gn​t and gr reveals that for the 20M model the batch gradient similarity mostly comes from random collisions with task direction, with a similarity score of 0.10±0.09, while for larger models, gn​t is almost orthogonal to gr, with 7.58×10−5±0.02 for the 1B model, suggesting little to no gradient interference on this set of neurons.

5 Discussion

We develop a data-centric account of why larger models can learn tasks that smaller models fail to learn. Specifically, we show that larger models can learn rare tasks from the data mixture, and this phenomenon is explained by learning dynamics, i.e., competition of resources and retention of memories, as well as the task frequency and complexity. Our perspective highlights that understanding scaling requires thinking beyond model expressivity. We need to understand how learning dynamics are at play with task frequency and complexity. It also points toward more intentional design of data mixtures to better elicit target capabilities. For example, simply scaling up the frequency of a target task might provide a more efficient way to learn the task than scaling up the model size. Lastly, our findings on how better retention of memories enables learning rare tasks offers a new perspective that views memorization as a mechanism that can support learning abstraction: by retaining task instances longer, models can accumulate signals across batches to learn more generalizable structures of the task. This suggests memorization can in fact be beneficial, in line with arguments by Feldman [70].

Limitations

As noted above, our account for why larger models learn more emphasizes the interplay of learning dynamics, task frequency, and task complexity. However, as discussed in Sec. 1, there are other plausible accounts for explaining this phenomenon, e.g., ones that focus on model expressivity and sample efficiency. Our explanation hence should not be interpreted as a complete account of scaling. Instead, these explanations are complementary: expressivity constrains what can be represented, sample efficiency shapes how effectively data is used, and our account highlights how learning dynamics interact with the frequency and complexity of tasks. A full understanding likely requires accommodating all these explanations, rather than viewing them as competing hypotheses. We also note we validated our key theoretical results using the OLMo pre-training pipeline, finding the empirical results on the injected tasks strongly match what the theoretical results predict. However, we acknowledge that empirical validation in a realistic pre-training setting could still leave some analytic gaps. For example, we did not empirically verify behavior of larger-scale language models or over-trained language models. We also selected injected tasks that matched the frequency of tasks learned in OLMo pre-training, which does not rule out other scaling behaviors with extreme task frequency. Our empirical results should therefore be viewed as supporting evidence. We encourage future work to explore different training regimes, more tasks, and different frequency ranges.

Acknowledgments

The authors thank Blake Bordelon, Jacob Zavatone-Veth, and Core Francisco Park for several useful references that helped concretize the claims posited in this work, and Yasaman Bahri, Surya Ganguli, Ari Holtzman, Stephanie Chan, Freya Behrens, Tom McGrath, Owen Lewis, Atticus Geiger, Jack Merullo, and Thomas Fel for fruitful conversations during the course of this project. The authors also thank Thomas Icard for several comments on an earlier version of this draft. This research is supported in part by a grant from Open Philanthropy (Coefficient Giving) to CP.

References

• [1] Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267, 2025.
• [2] Anthropic. System Card: Claude Mythos Preview, 2026a. https://www-cdn.anthropic.com/08ab9158070959f88f296514c21b7facce6f52bc.pdf.
• [3] Google DeepMind. Gemini 3 Pro - Model Card, 2026. https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-Pro-Model-Card.pdf.
• [4] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
• [5] DeepSeek-AI. DeepSeek-V4-Pro, 2026. https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf.
• [6] Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026.
• [7] Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, et al. Measuring AI ability to complete long software tasks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
• [8] Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, et al. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai. arXiv preprint arXiv:2411.04872, 2024.
• [9] ARC Prize Foundation. ARC-AGI-3, 2026. https://arcprize.org/arc-agi/3.
• [10] Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=a7Qa4CcHak.
• [11] Anthropic. Responsible Scaling Policy, 2026b. https://www.anthropic.com/responsible-scaling-policy.
• [12] OpenAI. Our Approach to Frontier Risk, 2023. https://openai.com/global-affairs/our-approach-to-frontier-risk/.
• [13] Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508.10104, 2025.
• [14] Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, Zhiyuan Liu, and Maosong Sun. Predicting emergent abilities with infinite resolution evaluation. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=lDbjooxLkD.
• [15] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openreview.net/forum?id=yzkSU5zdwD.
• [16] Jason Wei. 137 emergent abilities of large language models, 2022. https://www.jasonwei.net/blog/emergence.
• [17] Sanjeev Arora and Anirudh Goyal. A theory for emergence of complex skills in language models. arXiv preprint arXiv:2307.15936, 2023.
• [18] Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=35DAviqMFo.
• [19] Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846, 2023.
• [20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
• [21] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017.
• [22] Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=ryenvpEKDr.
• [23] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020.
• [24] Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01293, 2021.
• [25] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021.
• [26] Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300–22312, 2022.
• [27] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
• [28] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=iBBcRUlOAPR.
• [29] Tim Pearce and Jinyeop Song. Reconciling kaplan and chinchilla scaling laws. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=NLoaLyuUUF.
• [30] Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scaling laws. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=nbOY1OmtRc.
• [31] Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. How feature learning can improve neural scaling laws. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=dEypApI1MZ.
• [32] Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27):e2311878121, 2024.
• [33] Licong Lin, Jingfeng Wu, Sham M. Kakade, Peter L. Bartlett, and Jason Lee. Scaling laws in linear regression: Compute, parameters, and data. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 60556–60606. Curran Associates, Inc., 2024. doi: 10.52202/079017-1937. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/6fcb1afcc1e9c2c82c8ddddf03bcf0f6-Paper-Conference.pdf.
• [34] Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 28699–28722. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/5b6346a05a537d4cdb2f50323452a9fe-Paper-Conference.pdf.
• [35] Alexander Maloney, Daniel A Roberts, and James Sully. A solvable model of neural scaling laws. arXiv preprint arXiv:2210.16859, 2022.
• [36] Ekdeep Singh Lubana, Kyogo Kawaguchi, Robert P. Dick, and Hidenori Tanaka. A percolation model of emergence: Analyzing transformers trained on a formal language. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=0pLCDJVVRD.
• [37] Francesco Cagnetta, Hyunmo Kang, and Matthieu Wyart. Learning curves theory for hierarchically compositional data with power-law distributed features. In Forty-second International Conference on Machine Learning, 2025a. URL https://openreview.net/forum?id=Lw0kC75dY0.
• [38] Francesco Cagnetta, Alessandro Favero, Antonio Sclocchi, and Matthieu Wyart. Scaling laws and representation learning in simple hierarchical languages: Transformers vs. convolutional architectures. arXiv preprint arXiv:2505.07070, 2025b.
• [39] Francesco Cagnetta, Allan Raventós, Surya Ganguli, and Matthieu Wyart. Deriving neural scaling laws from the statistics of natural language. arXiv preprint arXiv:2602.07488, 2026.
• [40] Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Pareto frontiers in deep feature learning: Data, compute, width, and luck. Advances in Neural Information Processing Systems, 36:48021–48034, 2023.
• [41] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Validad Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=i1uGbfHHpH.
• [42] Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802, 2026.
• [43] Bloomberg. OpenAI Claims DeepSeek Distilled US Models to Gain an Edge, 2026. https://www.bloomberg.com/news/articles/2026-02-12/openai-accuses-deepseek-of-distilling-us-models-to-gain-an-edge?
• [44] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
• [45] Huajian Xin, Z.Z. Ren, Junxiao Song, Zhihong Shao, Wanjia Zhao, Haocheng Wang, Bo Liu, Liyue Zhang, Xuan Lu, Qiushi Du, , et al. Deepseek-prover-v1.5: Harnessing proof assistant feedback for reinforcement learning and monte-carlo tree search. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=I4YAIwrsXa.
• [46] Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=3zKtaqxLhW.
• [47] Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026.
• [48] Yunhao Tang, Sid Wang, Lovish Madaan, and Remi Munos. Beyond verifiable rewards: Scaling reinforcement learning in language models to unverifiable data. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum?id=pc6M9h3T9m.
• [49] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025.
• [50] Cody Blakeney, Jessica Zosa Forde, Jonathan Frankle, Ziliang Zong, and Matthew L Leavitt. Reduce, reuse, recycle: Improving training efficiency with distillation. arXiv preprint arXiv:2211.00683, 2022.
• [51] Shikai Qiu, Lechao Xiao, Andrew Gordon Wilson, Jeffrey Pennington, and Atish Agarwala. Scaling collapse reveals universal dynamics in compute-optimally trained neural networks. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=Fvq9ogLnLN.
• [52] Elliot Paquette, Courtney Paquette, Lechao Xiao, and Jeffrey Pennington. 4+3 phases of compute-optimal neural scaling laws. Advances in Neural Information Processing Systems, 37:16459–16537, 2024.
• [53] Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 OLMo 2 Furious, 2024. URL https://arxiv.org/abs/2501.00656.
• [54] Yedi Zhang, Andrew M Saxe, and Peter E. Latham. Saddle-to-saddle dynamics explains a simplicity bias across neural network architectures. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=Vit5M0G5Gb.
• [55] Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics. In The Thirty Sixth Annual Conference on Learning Theory, pages 2552–2623. PMLR, 2023.
• [56] Arthur Jacot, François Ged, Berfin Şimşek, Clément Hongler, and Franck Gabriel. Saddle-to-saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity. arXiv preprint arXiv:2106.15933, 2021.
• [57] Daniel Kunin, Giovanni Luca Marchetti, Feng Chen, Dhruva Karkada, James B Simon, Michael R DeWeese, Surya Ganguli, and Nina Miolane. Alternating gradient flows: A theory of feature learning in two-layer neural networks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum?id=t7LKc0MMW6.
• [58] Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Guha Thakurta, Nicolas Papernot, and Chiyuan Zhang. Measuring forgetting of memorized training examples. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=7bJizxLKrR.
• [59] Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267–284, Santa Clara, CA, August 2019. USENIX Association. ISBN 978-1-939133-06-9. URL https://www.usenix.org/conference/usenixsecurity19/presentation/carlini.
• [60] Jing Huang, Diyi Yang, and Christopher Potts. Demystifying verbatim memorization in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10711–10732, 2024.
• [61] Johnny Wei, Ameya Godbole, Mohammad Aflah Khan, Ryan Yixiang Wang, Xiaoyuan Zhu, James Flemings, Nitya Kashyap, Krishna P. Gummadi, Willie Neiswanger, and Robin Jia. Hubble: a model suite to advance the study of LLM memorization. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=ZfdnZhOP0k.
• [62] Hyeonbin Hwang and Yeachan Park. Intrinsic task symmetry drives generalization in algorithmic tasks, 2026. URL https://arxiv.org/abs/2603.01968.
• [63] Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arXiv preprint, 2024. URL https://huggingface.co/datasets/allenai/dolma.
• [64] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. OLMo: Accelerating the science of language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15789–15809, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.841. URL https://aclanthology.org/2024.acl-long.841/.
• [65] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022.
• [66] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, sep 2022. URL https://openreview.net/forum?id=9XFSbDPmdW.
• [67] Tianyi Zhou, Deqing Fu, Vatsal Sharan, and Robin Jia. Pre-trained large language models use fourier features to compute addition. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=i4MutM2TZb.
• [68] Sheridan Feucht, Tal Haklay, Usha Bhalla, Daniel Wurgaft, Can Rager, Raphaël Sarfati, Jack Merullo, Thomas McGrath, Owen Lewis, Ekdeep Singh Lubana, Thomas Fel, and Atticus Geiger. Arithmetic in the wild: Llama uses base-10 addition to reason about cyclic concepts, 2026. URL https://arxiv.org/abs/2605.01148.
• [69] Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. Finding alignments between interpretable causal variables and distributed neural representations. In Francesco Locatello and Vanessa Didelez, editors, Proceedings of the Third Conference on Causal Learning and Reasoning, volume 236 of Proceedings of Machine Learning Research, pages 160–187. PMLR, 01–03 Apr 2024. URL https://proceedings.mlr.press/v236/geiger24a.html.
• [70] Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd annual ACM SIGACT symposium on theory of computing, pages 954–959, 2020.
• [71] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020.
• [72] Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36:34201–34227, 2023.
• [73] Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36, 2024.
• [74] Sang Michael Xie. Foundation Models from a Data-Distributional View. Stanford University, 2024.
• [75] Rahul Ramesh, Jialin Mao, Itay Griniasty, Rubing Yang, Han Kheng Teoh, Mark Transtrum, James P Sethna, and Pratik Chaudhari. A picture of the space of typical learnable tasks. arXiv preprint arXiv:2210.17011, 2022.
• [76] Rahul Ramesh. The Principles of Learning on Multiple Tasks. PhD thesis, University of Pennsylvania, 2025.
• [77] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon LLM: Outperforming curated corpora with web data only. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum?id=kM5eGcdCzq.
• [78] Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:30811–30849, 2024.
• [79] Pratyush Maini, Vineeth Dorna, Parth Doshi, Aldo Carranza, Fan Pan, Jack Urbanek, Paul Burstein, Alex Fang, Alvin Deng, Amro Abbas, et al. Beyondweb: Lessons from scaling synthetic data for trillion-scale pretraining. arXiv preprint arXiv:2508.10975, 2025.
• [80] Dylan Sam, Sachin Goyal, Pratyush Maini, Alexander Robey, and J Zico Kolter. When should we introduce safety interventions during pretraining? arXiv preprint arXiv:2601.07087, 2026.
• [81] Sachin Goyal, Pratyush Maini, Zachary C Lipton, Aditi Raghunathan, and J Zico Kolter. Scaling laws for data filtering–data curation cannot be compute agnostic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22702–22711, 2024.
• [82] Rich Caruana. Multitask learning. Machine learning, 28(1):41–75, 1997.
• [83] Rahaf Aljundi. Continual learning in neural networks. arXiv preprint arXiv:1910.02718, 2019.
• [84] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in neural information processing systems, 34:18878–18890, 2021.
• [85] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in neural information processing systems, 32, 2019.
• [86] Runzhe Wu, Ankur Samanta, Ayush Jain, Scott Fujimoto, Jeongyeol Kwon, Ben Kretzu, Youliang Yu, Kaveh Hassani, Boris Vidolov, and Yonathan Efroni. Imbalanced gradients in rl post-training of multi-task llms. In Findings of the Association for Computational Linguistics: EACL 2026, pages 3137–3150, 2026.
• [87] Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. Advances in Neural Information Processing Systems, 34:1256–1272, 2021.
• [88] Itay Evron, Edward Moroshko, Rachel Ward, Nathan Srebro, and Daniel Soudry. How catastrophic can catastrophic forgetting be in linear regression? In Conference on Learning Theory, pages 4028–4079. PMLR, 2022.
• [89] Martin Marek, Dongkyu Cho, Shikai Qiu, Rumi Chunara, Pavel Izmailov, and Andrew Gordon Wilson. Forgetting in language models: Capacity, optimization, and self-generated replay, 2026. URL https://arxiv.org/abs/2605.26097.
• [90] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in neural information processing systems, 33:5824–5836, 2020.
• [91] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31, 2018.
• [92] Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pages 794–803. PMLR, 2018.
• [93] Mihai Suteu and Yike Guo. Regularizing deep multi-task networks using orthogonal gradients. arXiv preprint arXiv:1912.06844, 2019.
• [94] Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International conference on artificial intelligence and statistics, pages 3762–3773. PMLR, 2020.
• [95] Peter L Chen, Xiaopeng Li, Xi Chen, and Tianyi Lin. Reward-free alignment for conflicting objectives. arXiv preprint arXiv:2602.02495, 2026.
• [96] Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=GhVS8_yPeEa.
• [97] Darshil Doshi, Tianyu He, Aritra Das, and Andrey Gromov. Grokking modular polynomials. arXiv preprint arXiv:2406.03495, 2024.
• [98] Pulkit Gopalani, Ekdeep S Lubana, and Wei Hu. Abrupt learning in transformers: A case study on matrix completion. Advances in Neural Information Processing Systems, 37:55053–55085, 2024.
• [99] Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher Manning. Grokking of hierarchical structure in vanilla transformers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 439–448. Association for Computational Linguistics, July 2023. URL https://aclanthology.org/2023.acl-short.38/.
• [100] Tanishq Kumar, Blake Bordelon, Samuel J. Gershman, and Cengiz Pehlevan. Grokking as the transition from lazy to rich training dynamics. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=vt5mnLVIVo.
• [101] Dashiell Stander, Qinan Yu, Honglu Fan, and Stella Biderman. Grokking group multiplication with cosets. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=hcQfTsVnBo.
• [102] Mohamad Amin Mohamadi, Zhiyuan Li, Lei Wu, and Danica J. Sutherland. Why do you grok? a theoretical analysis on grokking modular addition. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=ad5I6No9G1.
• [103] Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar. Explaining grokking through circuit efficiency. arXiv preprint arXiv:2309.02390, 2023.
• [104] Depen Morwani, Benjamin L. Edelman, Costin-Andrei Oncescu, Rosie Zhao, and Sham M. Kakade. Feature emergence via margin maximization: case studies in algebraic tasks. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=i9wDX850jR.
• [105] Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew L Leavitt, and Naomi Saphra. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=MO5PiKHELW.
• [106] Chen Cheng, John Duchi, and Rohith Kuditipudi. Memorize to generalize: on the necessity of interpolation in high dimensional linear regression. In Conference on Learning Theory, pages 5528–5560. PMLR, 2022.
• [107] Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. When is memorization of irrelevant training data necessary for high-accuracy learning? In Proceedings of the 53rd annual ACM SIGACT symposium on theory of computing, pages 123–132, 2021.
• [108] Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4):667–766, 2022.
• [109] Marco Loog, Tom Viering, Alexander Mey, Jesse H Krijthe, and David MJ Tax. A brief prehistory of double descent. Proceedings of the National Academy of Sciences, 117(20):10625–10626, 2020.
• [110] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=B1g5sA4twr.
• [111] Preetum Nakkiran. More data can hurt for linear regression: Sample-wise double descent. arXiv preprint arXiv:1912.07242, 2019.
• [112] Preetum Nakkiran, Prayaag Venkat, Sham M. Kakade, and Tengyu Ma. Optimal regularization can mitigate double descent. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=7R7fAoUygoa.
• [113] Daniel Wurgaft, Ekdeep Singh Lubana, Core Francisco Park, Hidenori Tanaka, Gautam Reddy, and Noah Goodman. In-context learning strategies emerge rationally. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum?id=bBUUOQI0N6.
• [114] Aaditya K Singh, Ted Moskovitz, Felix Hill, Stephanie C.Y. Chan, and Andrew M Saxe. What needs to go right for an induction head? a mechanistic study of in-context learning circuits and their formation. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=O8rrXl71D5.
• [115] Aaditya Singh, Stephanie Chan, Ted Moskovitz, Erin Grant, Andrew Saxe, and Felix Hill. The transient nature of emergent in-context learning in transformers. Advances in neural information processing systems, 36:27801–27819, 2023.
• [116] Core Francisco Park, Ekdeep Singh Lubana, and Hidenori Tanaka. Competition dynamics shape algorithmic phases of in-context learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=XgH1wfHSX8.
• [117] Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In International conference on machine learning, pages 15696–15707. PMLR, 2023.
• [118] Pietro Lesci, Clara Meister, Thomas Hofmann, Andreas Vlachos, and Tiago Pimentel. Causal estimation of memorisation profiles. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15616–15635, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.834. URL https://aclanthology.org/2024.acl-long.834/.
• [119] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=TatRHT_1cK.
• [120] Rohith Kuditipudi, Jing Huang, Sally Zhu, Diyi Yang, Christopher Potts, and Percy Liang. Blackbox model provenance via palimpsestic membership inference. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum?id=VRhVS59yhP.
• [121] Dmitrii Krasheninnikov, Richard E. Turner, and David Krueger. Fresh in memory: Training-order recency is linearly encoded in language model activations. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=Tn6famjSxN.
• [122] Sunny Duan, Mikail Khona, Abhiram Iyer, Rylan Schaeffer, and Ila R Fiete. Uncovering latent memories in large language models. In The Thirteenth International Conference on Learning Representations, 2025.
• [123] Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang, Youngkyung Seo, Du-Seong Chang, and Minjoon Seo. How do large language models acquire factual knowledge during pretraining? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=TYdzj1EvBP.
• [124] Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274–38290, 2022.
• [125] Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from repeated data. arXiv preprint arXiv:2205.10487, 2022.
• [126] Steven T Piantadosi. Zipf’s word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 21(5):1112–1130, 2014.
• [127] Aapo Hyvärinen, Jarmo Hurri, and Patrick O Hoyer. Natural image statistics: A probabilistic approach to early computational vision., volume 39. Springer Science & Business Media, 2009.
• [128] Alexander Atanasov, Jacob A Zavatone-Veth, and Cengiz Pehlevan. Scaling and renormalization in high-dimensional regression. arXiv preprint arXiv:2405.00592, 2024.
• [129] Yunwei Ren, Eshaan Nichani, Denny Wu, and Jason D. Lee. Emergence and scaling laws in SGD learning of shallow neural networks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum?id=kA2H90nm26.
• [130] Katie Everett, Lechao Xiao, Mitchell Wortsman, Alexander A Alemi, Roman Novak, Peter J Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jaehoon Lee, et al. Scaling exponents across parameterizations and optimizers. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=0ksNeD1SJT.
• [131] Eric J Michaud, Liv Gorton, and Tom McGrath. Understanding sparse autoencoder scaling in the presence of feature manifolds. arXiv preprint arXiv:2509.02565, 2025.
• [132] Yoonsoo Nam, Nayara Fonseca, Seok H Lee, Chris Mingard, and Ard A Louis. An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem. Advances in Neural Information Processing Systems, 37:39632–39693, 2024.
• [133] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=rJl-b3RcF7.
• [134] Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In International conference on machine learning, pages 6682–6691. PMLR, 2020.
• [135] Ankit Pensia, Shashank Rajput, Alliot Nagle, Harit Vishwakarma, and Dimitris Papailiopoulos. Optimal lottery tickets via subset sum: Logarithmic over-parameterization is sufficient. Advances in neural information processing systems, 33:2599–2610, 2020.
• [136] Ian Magnusson, Nguyen Tai, Ben Bogin, David Heineman, Jena D. Hwang, Luca Soldaini, Akshita Bhagia, Jiacheng Liu, Dirk Groeneveld, Oyvind Tafjord, Noah A. Smith, Pang Wei Koh, and Jesse Dodge. Datadecide: How to predict best pretraining data with small experiments. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=p9YlQPF8fE.
• [137] Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. In First Conference on Language Modeling, 2024b. URL https://openreview.net/forum?id=u2vAyMeLMm.
• [138] Zhengxuan Wu, Atticus Geiger, Aryaman Arora, Jing Huang, Zheng Wang, Noah Goodman, Christopher Manning, and Christopher Potts. pyvene: A library for understanding and improving PyTorch models via interventions. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: System Demonstrations), pages 158–165. Association for Computational Linguistics, June 2024. URL https://aclanthology.org/2024.naacl-demo.16.
• [139] Ky Fan. On a theorem of weyl concerning eigenvalues of linear transformations i. Proceedings of the National Academy of Sciences, 35(11):652–655, 1949.
• [140] Nikhil Vyas, Alexander Atanasov, Blake Bordelon, Depen Morwani, Sabarish Sainathan, and Cengiz Pehlevan. Feature-learning networks are consistent across widths at realistic scales. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=LTdfYIvbHc.

Appendix A Related Work

Multi-Task Learning.

Data distributions neural networks are trained on are often deemed as a mixture of tasks [71, 72, 73, 74, 75, 76, 77, 78, 63, 79, 80, 81]. This motivated works analyzing both the learning dynamics of training toy models on multi-task distributions and defining methods aimed at reducing interference between updates caused by learning a task in the presence of other ones. For example, the notion of “catastrophic interference” has been often characterized in the multi-task learning and continual learning literature [82, 83], where task gradients conflict or are imbalanced in scale, leading to learning of only a subset of tasks instead of the entire mixture. Such phenomenology can be intuitively [84, 85, 86] and theoretically explained: e.g., Pezeshki et al. [87] posit the idea of gradient starvation, whereby a model trained on a mixture of tasks that have different prior frequencies is unable to learn the infrequent task due to its gradient getting “starved” out, i.e., becoming zero; meanwhile, Evron et al. [88] characterize how tasks’ observation frequency induces the forgetting of another learned task in a sequential linear regression setting. Concurrent to our work, Marek et al. [89] show that forgetting of prior tasks occur when a model has little remaining capacity. These analyses have also motivated methods to avoid interference and enable learning of multiple tasks: e.g., methods that perform “surgery” on model gradients [90, 91, 92] to make two conflicting tasks’ gradients to have zero interaction by removing one’s projection towards another [93, 94]; these methods have seen use at scale as well [95].

It is worth noting that our results are in a similar vein as literature above, but augment prior work by characterizing the effects of scale and showcasing that even extremely rarely observed tasks can eventually be learned if one’s model is large enough—empirically, related results corroborating our claim in a vision scenario was also made in the continual learning literature by Ramasesh et al. [96]. That said, we emphasize that neither do our results imply nor do we claim (in fact, we say otherwise) that scale alone is the mechanism to enable the learning of a rare task in the presence of other frequent ones. Indeed, methods discussed above from multi-task / continual learning literature show multiple tasks can be simultaneously learned by a model.

Memorization and Scaling.

The core mechanism posited in our work for how larger models learn rare tasks involves a model retaining some signature of observed data from a small batch of samples. In extreme scenarios, e.g., when only a few samples are contained in the batch, such a signature cannot possibly correspond to a general, abstract task representation. Instead, the signature can be thought of as a model (at least partially) “memorizing” an observation—once enough observations occur and the memories aggregate, in our simple toy settings, we find the model consolidates the memories into an abstract representation that generalizes well. In this sense, we emphasize our core proposition suggests memorization is not an undesirable property, but instead a prerequisite to eventual generalization for rare tasks. This mechanism is highly reminiscent of the observations [65, 97, 98] and posited learning dynamics for grokking, where a model transitions from memorizing observations to generalizing to novel inputs [66, 99, 100, 101, 102, 103, 104]. Critically, our language model pretraining results, where we use modular addition, i.e., the prototypical grokking task, and find our posited dynamics hold is suggestive that grokking-like dynamics may in fact occur in practice, especially for rarely observed tasks (closest result to this end is perhaps the syntax acquisition dynamics demonstrated by Chen et al. [105]). This argument is in keeping with theoretical works on classification that have argued that memorization is necessary for generalization—for example, to handle label noise [106], or to handle rare examples [70]. Brown et al. [107] provide a particularly interesting demonstration that learning rare structures effectively requires memorizing even irrelevant information about the data. On the other hand, it is worth considering if an opposite mechanism may occur for frequently observed tasks: e.g., if a model sees too many observations of the same task, does it perhaps undergo phenomenology such as overfitting, which is generally associated with generalization to memorization dynamics; if so, does scaling help avoid this dynamic via mechanisms such as double descent [108, 109, 110, 111, 112]? Recent work on transient nature of in-context learning capabilities in toy scenarios [113, 114, 115, 116] is suggestive such a dynamic may occur, and we thus argue it is worth investigating what the counterpart of our work for learning dynamics of frequent tasks looks like.

Building on the above, we also note memorization and the effects of scaling have been often studied in literature; these results are in line with our claims on reduced interference over model parameters via scaling, enabling models to eventually learn rare tasks. For example, studying memorization in the sense of verbatim match (e.g., k-token string match), works show larger models learn knowledge present in the tail-end of the distribution better [117], larger models [118, 119] and later checkpoints tend to memorize more [60], not just individual data points but also training data order [120, 121], and these memories are retained for longer across injection events [122, 123]. Tirumala et al. [124] show that larger models memorize more, but also can memorize more of the data before they begin to overfit.

Generalization and Scaling.

Improved performance as a function of scaling has defined the spirit of machine learning since scaling laws first started being used for identifying training configurations [21, 22, 20, 25, 28, 23, 125, 24]. The precise mechanism as to how scaling helps produce better models is unclear, but a few propositions have been made. For example, in works assessing how power-law scaling as a function of data and parameters emerges, prior work has exploited the argument that natural data statistics are heavy-tailed and follow power-law trends (e.g., Zipf-priors in language [126] and vision [127]); correspondingly, scaling enables access to lower-order modes of the data distribution [32, 35, 33, 128, 129, 37, 38], and arguments to this end have been verified in recent work by Cagnetta et al. [39]. Our toy setup was in fact inspired by these papers, especially Ren et al. [129] and Maloney et al. [35], but is a substantial simplification since, unlike these prior works, our goal was not to characterize the eventual steady state optima a model arrives at, but instead the dynamics that lead to it. Works closer to this dynamical motivation are by Bordelon et al. [30, 31], Paquette et al. [52], Atanasov et al. [128], and Everett et al. [130], who analyze learning dynamics of toy settings that exhibit power-law scaling curves. However, since we primarily aimed to posit a concrete mechanism via which larger models may be able to learn tasks smaller models do not, we note the concrete results emphasized and takeaways across these works versus ours are fairly different. In particular, these papers primarily focus on the interaction between learning dynamics and data statistics to identify different regimes of scaling, i.e., what functional form, e.g., power-law or otherwise, results in the best effective characterization of learning dynamics. Finally, works by Michaud et al. [34, 131] and Nam et al. [132] are fairly related to our paper: specifically, these works characterize an explicitly multi-task construction to posit a model for how power-law scaling can emerge in neural networks. While still related to the data statistics argument mentioned above, these works also have an explicit notion of prior frequency and (implicitly) show scaling helps learn tasks that are rarely observed in the training distribution. Our work makes this claim explicit, but also characterizes how, i.e., a mechanism, via which scaling aids learning of rare tasks.

Lottery Tickets and Scaling.

Another thread of research that partially connects the work listed above on scaling and learning of specific tasks is on the lottery ticket hypothesis [133]: a lottery ticket is defined as a subnetwork identified from a larger, initial network that, even at random initialization, shows the ability to perform the task one is training their model for. Theoretical work on lottery ticket hypothesis has characterized bounds on how much larger a model has to be in order to possess a subnetwork that can, up to some error, approximate the model eventually learned via training [134, 135, 40]. Especially related here is the work of Edelman et al. [40], who show that via scaling model width (the scaling axis we consider as well), the odds that a subset of representations with non-trivial alignment with true task features exist substantially increases. Correspondingly, scaling improves sample efficiency of learning tasks that require more features (i.e., are more complex); critically, if one slightly generously interprets the authors’ results, they are suggestive that a larger model will be able to learn rare tasks by virtue of already possessing features a smaller model will be unable to learn (due to sparsely observed training signal for such tasks). While this work partially informed the intuition guiding this paper, we note the eventual results for our setting and verification on large-scale scenarios are more concrete.

Appendix B Experimental Details

B.1 Synthetic Experiment

In the following, we describe experiment details and metrics relevant to results for the synthetic setup.

Data-generating process.

All synthetic runs use the orthogonal-block instantiation of the mixture-of-regressions setup proposed in Sec. 3. We fix the ambient dimension at D=1024, the number of tasks at K∈{16,32} (almost all figures use K=32), and a per-task block dimension dT such that K⋅dT≤D and the task blocks are mutually orthogonal. Concretely, task k occupies coordinates [k​dT,(k+1)​dT) of ℝD, and its within-block spectrum is the power-law σk,j=j−αk for j=1,…,dT. Unless stated otherwise we use a shared exponent αk≡α across tasks (α=1 in the orthogonal-block experiments, making the within-block decay slow enough that capacity reliably spreads beyond the leading mode of each task). The task prior is the power-law πk∝k−β, normalized to sum to one over k=1,…,K; β=2 in most experiments. Inputs are sampled fresh each step as x∼𝒩⁡(0,σin2​ID) with σin=1 in all orthogonal-block runs. The per-task targets are yk=Λk1/2​Bk⊤​x, restricted to the task’s block; because each task block has rank dT, the output dimension of the regressor is dT (and reduces to 1 when dT=1, e.g., in the rank-1 specialization of App. E.1.2).

Model.

The student is the linear-bottleneck regressor of Sec. 3: a shared encoder W∈ℝN×D that maps the input to an N-dimensional hidden, followed by per-task linear decoders Dk∈ℝdT×N selected by the ground-truth task index supplied in the batch. We do not explicitly constrain W to have orthonormal rows: the relevant object for Theorem 3 is the projector PW=W⊤​(W​W⊤)−1​W, which is invariant to the right-multiplicative gauge of W and which gradient flow drives toward the top-N eigenspace of M=∑kπk​Ck regardless of the parametrization. The encoder is initialized such that W⊤​W=IN at step zero, and the per-task decoders are initialized with Kaiming-uniform fan-in / linear gain. The decoders are jointly optimized with the encoder rather than analytically closed-formed at each step, since learned decoders will converge to Dk∗=Λk1/2​Bk⊤​U at any stationary point, so the joint optimization does not change the encoder fixed point but does match the practical setting in which both ends of the bottleneck are learned simultaneously.

Optimizer.

We use AdamW with default hyperparameters and an inverse-square-root learning-rate schedule. Gradients are clipped at maximum norm 1.0. Batches are drawn fresh each step (no fixed dataset, no replay) with batch size B=1 024 for the phase-diagram and rank-1 sweeps, and B=512 for the matched-frequency retention sweeps; the smaller batch in the retention runs is required so that an injection batch with m≤B rare-task slots can match the long-run frequency ρr=m/(G⋅B) at the ρr≈6×10−4 end of the sweep.

Metrics.

We track three families of metrics, all reported on freshly sampled batches separate from the training stream. The first is the per-task loss, i.e., the unnormalized population MSE ℓk​(U)=𝔼⁡[‖yk−Dk​U⊤​x‖22], and its normalized counterpart ℓk​(U)/ℓk,baseline with ℓk,baseline=‖ak‖22/dT the mean-predictor MSE per task. The second is the per-task subspace alignment, the basis-free quantity sk​(U)=Tr⁡(PU​Ck)/Tr⁡(Ck)=‖PU​ak‖22/‖ak‖22, computed via the SVD of W so that it is independent of the gauge of the encoder. sk lies between N/D at random initialization and 1 when the task block is fully captured. We also report its random-baseline-corrected normalization s~k​(U)=(sk​(U)−N/D)/(1−N/D), which equals 0 at random initialization and 1 at full capture. The third is the residual common-task signal: we compute δF​(U)=∑k∈Fπk​(1−sk​(U))​‖ak‖22, the residual energy of the frequent block. The frequent set F is the smallest top-prior set with cumulative mass at least 0.8; under our power-law prior this yields |F|={6,3,2,2} for β∈{0.5,1.0,1.5,2.0} respectively. Standard evaluation is performed every 1 000–2 000 steps on a held-out probe of the same population distribution, with the final checkpoint additionally re-evaluated for end-of-training summary statistics.

B.2 OLMo Pretraining Pipeline

Models.

We use the OLMo model architecture [64]. For 4M to 1B models, we follow the model configuration and naming convention of Magnusson et al. [136]. We additionally include a 4B model to further evaluate width scaling, as shown in Table 1.

Training hyperparameters.

We use the same batch size of 1024, window size of 4096 for TCMP and 1024 for TADD, and a learning rate schedule with an initial learning rate of 3×10−4 and cosine with warmup schedule for all models. For the retention window ablation experiment, we use a smaller window size of 512 to reduce the training cost. For a full list of hyperparameters, refer to the OLMo-7B-0724 configuration.⁵⁵ 5 https://github.com/allenai/OLMo/blob/main/configs/official-0724/OLMo-7B-0724.yaml

Training pipeline.

We use the OLMo code base.⁶⁶ 6 https://github.com/allenai/OLMo Our usage is in line with its Apache-2.0 license.⁷⁷ 7 https://github.com/allenai/OLMo?tab=Apache-2.0-1-ov-file

Compute resources.

All models are trained on a cluster of NVIDIA H200 GPUs.

B.3 Pre-training and Injected Task Data

Pre-training data.

We use Dolma v1.7 as the pre-training corpus [63]. Specifically, we use the 210B tokens corresponding to the first 50K batches that OLMo-7B-0424 and OLMo-7B-0724 are trained on, in the exact same order.

Reference tasks.

To ensure the injected task frequency is comparable to the frequency of tasks learned in pre-training, we sample two reference tasks Rcmp and Radd from pre-training that involve similar high-level functions. Rcmp predicts a number larger than x in the prompt “it has increased from {x} to”. Radd predicts the sum of two numbers smaller than 100 with the prompt “{x} + {y} =”. We estimate the lower bound of their frequency in pre-training data using infini-gram [137] and observe models’ next token prediction loss on the task, which corresponds to the two dashed lines in Fig. 6 panel (a).

Injected tasks.

We elaborate the task label here. Let v​a​l​(⋅):𝒮↦[0,99] be a bijective mapping that assigns an integer value between 0 and 99 to each token. For TCMP, LABEL is one of two tokens randomly chosen from the vocab indicating whether v​a​l​(TOK1)<v​a​l​(TOK2). For TADD, LABEL is the token in 𝒮 whose value equals (v​a​l​(TOK1)+v​a​l​(TOK2)) m​o​d 100. Below are a few instances from the comparison task: address analyze pony, resort zebrafish pony, cavities misconduct provisional, where pony and provisional are the two label tokens that represent True and False.

B.4 Localizing and Measuring Task Features in Sec. 4.3

The comparison task TCMP.

We first use distributed alignment search (DAS) to verify that the model’s prediction is causally dependent on a global token order feature, which is encoded in a 1-D subspace in the residual stream of the first few layers.

The task has a simple high-level causal model, namely X→O←Y, where X,Y are the two inputs and O is the binary output. We consider the following intervention on the input variable X (or Y): Let a base example be xb,yb,ob and a source example be xs,ys,os, an interchange intervention on X that sets the value of xb to xs should lead to a counterfactual label that corresponds to xs<yb.

In the neural model, we search for a low-dimensional subspace in the residual stream that plays the same causal role as the input variable X by training on 1​K counterfactual data pairs defined above. We use DAS to search across all layers above the input token position. We are able to find a 1-D subspace in the residual stream of the first layer that has an interchange intervention success rate of 96%. This proves that the model not only encodes the global token order in a low-dimensional space but actually uses this feature for prediction on the task TCMP. This allows us to use the global order feature to measure to what extent a model has learned the abstract task structure.

We use the DAS implementation from pyvene [138].

The modular addition task TADD.

As prior work studying modular addition has identified that grokked models use Fourier modes for addition [66], we conduct Fourier analysis on residual stream to measure the presence of Fourier modes.

For modulus P, define a real discrete Fourier transform basis on ℝP as follows:

At layer l, collect residual stream vectors hl∈ℝd grouped by output c = (a+b) mod P and compute the mean representation as:

This yields a matrix V(l)∈ℝP×d.

After row-centering V(l), the fraction of variance captured by frequency k is

We consider a null-baseline for detecting Fourier modes in representations. Under uniform variance allocation across frequencies,

Hence, a frequency k at layer l is identified as a Fourier mode if

We choose θ=2 in our experiment, as we do not observe significant difference in grokking behavior for models that represent Fourier modes with stronger signals, e.g., θ=3.

Finally, we define the total number of detected modes as follows, where L is the total number of layers.

This corresponds to the y-axis in Fig. 7 (b) right panel.

B.5 Localizing Task Neurons in Sec. 4.4

We conduct null interventions on MLP layers to identify layers that have the largest causal effects on the model output. The results are shown in Fig. 10. This aligns with our observation from the DAS localization experiment that first layer is the earliest layer where the global token order is causally encoded. The localization result is consistent across the three models used in Sec. 4.4.

Appendix C Proofs

C.1 Proof of Theorem 3

For fixed U, the task-k population loss is

where the second identity uses x∼𝒩⁡(0,I) and the standard relation 𝔼​‖A​x‖22=‖A‖F2. This is a linear least-squares problem in Dk, so the minimizer is

Substituting back gives

Summing with weights πk yields

Because M is symmetric positive semidefinite with finite trace, minimizing LN​(U) is equivalent to maximizing Tr⁡(U⊤​M​U) over all orthonormal U. By Ky Fan’s maximum principle [139],

where μ1≥μ2≥⋯ are the eigenvalues of M. Therefore any minimizer spans the top-N eigenspace of M and the optimal loss is

For our generative process, we have

so the vectors bk,j are eigenvectors of M with eigenvalues uk,j=πk​λk,j. Thus the width-N optimum keeps the N largest utilities. If task k contributes nk​(N) retained coordinates, then its residual loss is

C.2 Proof of Theorem 4

Write G𝖥​(U)=2​(I−PU)​M𝖥1/2​M𝖥1/2​U. Using ‖A​B‖F≤‖A‖F​‖B‖op, where ∥.∥op denotes the operator norm, gives

C.3 Proof of Proposition 6

Let U denote the common-task width-N solution and let ui be one occupied common eigenvector with eigenvalue μi𝖥. Replace that vector by

while keeping the remaining N−1 directions fixed. Because ui is an eigenvector of M𝖥 and br is orthogonal to the common block, the contribution of this one direction to the objective Tr⁡(PU​M) is

Subtracting the value at θ=0 gives

Since L=Tr⁡(M)−Tr⁡(PU​M), the loss change is

Hence perturbations toward br decrease the loss if and only if πr​λr>μi𝖥. The rare feature invades first through the weakest occupied common direction, which has curvature μN𝖥, proving the stated threshold.

C.4 Microscopic competition in a one-neuron, two-task model

Example 7(One neuron, two orthogonal tasks).

Let a,b∈ℝd be orthonormal and consider rank-one tasks with covariances Ca=a​a⊤ and Cb=b​b⊤. A width-1 encoder is a unit vector

The task losses are

A gradient step on task a obeys θ+=θ−η​sin⁡(2​θ)+O⁡(η2), while a step on task b obeys θ+=θ+η​sin⁡(2​θ)+O⁡(η2). If task a appears with probability p and task b with probability q<p, then

which drives the neuron toward the common task. Near θ=0, if a rare-task update is followed by G common-task updates, then

Thus rare-task alignment decays exponentially across the gap between rare observations.

Proof.

The loss identities follow from u⊤​a​a⊤​u=cos2⁡θ and u⊤​b​b⊤​u=sin2⁡θ. Differentiating yields

which gives the stated updates under gradient descent. Taking the expectation under the task mixture yields the drift formula. Linearizing sin⁡(2​θ)≈2​θ near zero gives the exponential decay estimate. ∎

The dynamics posited above are also exemplified in Fig. 11.

Appendix D Further Experimental Results: Complexity Sweeps

In the main paper, we kept “complexity”, i.e., the number of directions used for defining the target variable constant across tasks; specifically, tasks in the main paper require 5 directions to cover 90% of the energy in the task spectrum (i.e., solving for r in arg⁡minr⁡∑j=1j=rλk,j∑jλk,j>0.90 gives r=5). In this section, we vary this property by changing the power-law coefficient underlying the task spectrum, i.e., since λk,j∝j−αk, we vary the range of αk across tasks. We define ranges of [αmin,αmax], split the range uniformly into K values, and assign the kth value to αk. The most frequent task is assigned the value αmax, giving it the fastest decaying spectrum and hence making it the simplest task, while the rarest task is assigned the value αmin, giving it the slower decaying spectrum and making it most complex. In particular, we choose ranges (see Fig. 12) such that the task complexity varies between [4,7] and [2,12] across K=32 tasks.

D.1 Feature Utility Predicts Learning Order

We first reproduce Fig. 2. We split the figure into two parts, showing the critical width boundary as a function of task frequency in heatmaps in Fig. 13 and the per-task loss predictability based on feature utilities in Fig. 14; for reference, we include our baseline results from the main paper, where the task spectra were uniform.

D.2 Competition Dynamics Disallow Learning of the Rarest and the Most Complex Task

Our results above showed that learning trends under varying task complexity are modulated by both task frequency and complexity, as predicted by our account in Sec. 3. We now show the competition dynamics picture posited in that section continues to follow in these settings as well. In particular, we plot the learning of the top-3 most frequent tasks (measured by normalized signal; see Sec. 3 for details) and the rare-most task as a function of residual, i.e., signal remaining to be explained in the frequent tasks. As shown in Fig. 15, the critical width predicted to be necessary for learning of the rarest task, by rendering the residual sufficiently small for most frequent tasks, continues to hold true in this setup as well.

D.3 Reduced Interference Aids Learning of the Rarest and the Most Complex Task

We now validate our argument for how data-centric bottlenecks, i.e., the low-frequency and high-complexity nature of a task, is circumvented by a larger model: by virtue of having more parameters, a larger model witnesses reduced interference over per-task gradients. To this end, we redo the batch-injection experiments from Fig. 4 and plot the retention dynamics for the lowest aggregate utility task across settings. Results are shown in Fig. 16. We see similar results as before: larger models show better retention of observed signal from a task, allowing them to bootstrap on these past observations and eventually learn the task. Meanwhile, a medium width model is able to do so only when the task is observed sufficiently frequently, i.e., the gap is low. Comparing with the case when the width is too low, we see the model never learns the task and the retention dynamics concretely show why: the model is unable to retain signal for the observed task for long enough.

Appendix E Further Experimental Results: Frequency Sweeps

E.1 Features and Tasks are Learned in Order of Utility

E.1.1 Extending Phase Diagram

We sweep the power-law exponent β defining the task prior by varying it over {0.5,1.0,1.5,2.0}. Unlike Fig. 2, we only sweep 5 values of widths—specifically, N∈{8,16,32,64,128}. Correspondingly, the staircase is rendered at lower resolution. Results are shown in Fig. 17.

E.1.2 Simplified Case: Rank-1 Tasks

Theorem 3 predicts that a width-N minimizer retains the N task-features with largest utility uk​j=πk​λk​j. We obtain a sharp quantitative test of this claim by collapsing the orthogonal-block setup to its rank-1 specialization: setting dT=1 and αk=1 makes every task rank-1 with λk=1, so the utility ordering reduces to the prior ordering, and the predicted critical width for task k becomes

i.e., a perfectly linear staircase in task index.

Setup.

We train the linear-bottleneck student described in Sec. 3 on a mixture of K=32 rank-1 orthogonal tasks, ambient dimension D=1024, and a power-law prior with exponent β=2. We sweep the encoder width N∈{1,2,3,4,6,8,10,12,14,16,20,24,28,32,40,48,64} and read out the per-task subspace alignment sk​(U)=‖PU​bk‖2 at the end of training (rank-1 specialization of the per-task signal of Sec. 3, so sk​(U) lies between N/D at random initialization and 1 when bk is fully captured by the encoder subspace).

Result.

See Figure 18. We overlay the empirical transition width Nemp​(k)=min⁡{N∈grid:sk​(U)>0.5} on the theoretical staircase (27), finding almost perfect alignment (minimal disparities are an artifact of the sampled width grid).

E.2 Residual Controls Learning

We next provide further validation for results in Fig. 3 by repeating experiments across different values of exponents in the power-law task prior. Specifically we vary β from the set {0.5,1.0,1.5,2.0}. For each run we compute the per-task signal sk​(U)=Tr⁡(PU​Ck)/Tr⁡(Ck), i.e., how well the model representation encodes information about the kth task, and the residual δF​(U)=∑k∈Fπk​(1−sk​(U))​‖ak‖2 from the final checkpoint. The frequent set F is defined as the smallest set of tasks whose cumulative mass meets 0.8; under β∈{0.5,1,1.5,2} this yields |F|={6,3,2,2} respectively, reflecting how a flatter prior spreads the loss budget across more frequent tasks. Results are reported in Figure 19. We see a precise kink similar to Fig. 3 when rare-task signal drops to zero once δF exceeds the analytic threshold δF∗​(Nrcrit), and rises steeply to near-unity once δF falls below it. We also see the threshold itself shifts left as β grows: a steeper prior makes the rare task’s leading utility πr​λr much smaller, so a smaller residual is required to “free up” encoder directions that can then capture the rare task.

E.3 Per-gap dynamics: Reproducing retention results across different injection gaps and widths

We reproduce results from Fig. 4a by reporting the joint dynamics of the normalized rare-task signal s~r​(Ut) and its gain / decay dynamics as a function of rare-task injection events, as shown in Fig. 20. Similar to results seen in the main paper, we find larger models retain and preserve the learned signal, while smaller models require the gaps to be sufficiently small if learning is to occur at all.

E.4 Effects of Scaling Data: Learning Bottleneck Persists at Long Training Horizon

In the main paper, especially Sec. 2, we distinguish between finite vs. asymptotic training. However, most of our training runs use a budget of 100K training iterations. To contextualize that this budget is sufficient for the claims made in the paper, we extend training runs to 1M steps for 6 values of model widths N={8,16,32,64,128,256}, subsampling the range of widths analyzed in the main paper; the setup remains the same otherwise as Fig. 2. We find that results (see Fig. 21) are stable at much longer horizons: above-capacity tasks do not slowly close the gap given more training; instead they remain at or below the random-projection baseline indefinitely.

Appendix F Further Experimental Results in OLMo Setting

F.1 Task Loss vs. General Language Modeling Loss

Given the same language modeling loss, larger models can achieve lower task-specific loss.

We analyze the relationship between learning “frequent tasks” and learning the injected task. For evaluating “frequent tasks”, we measure the language modeling loss on the C4 validation set using a context window size of 256. As shown in Fig. 22, when the task frequency is relatively high, i.e., from 2.4×10−6 to 2.4×10−5, loss curves from all model sizes follow roughly the same trajectory. This suggests that in this frequency range, model size only improves sample efficiency, but these models still have similar training dynamics [140]. However, when the task frequency gets lower, i.e., 2.4×10−7, larger models achieve lower task loss given the same language modeling loss. Moreover, smaller models diverge from larger models one by one, with their injected-task loss plateauing at different values. This supports our hypothesis that for rare tasks, larger models have different learning dynamics that unlock the ability to learn rare tasks.

F.2 Compute-optimal Comparison

In Fig. 5, we compare models trained on the same amount of data. In Fig. 23, we further show that larger models are more compute-efficient at learning low-frequency tasks. When the frequency is one task instance per batch, i.e., 2.4×10−7, given the same compute budget, estimated as 6× the number of model parameters × the number of training tokens following Chinchilla scaling laws, larger models achieve lower task loss. Moreover, consistent with our observation in Fig. 22, smaller models initially follow the learning dynamics of larger models, but after a certain point their injected-task loss curves plateau and deviate from the larger models’ curves.