Transluce 研究微调效果预测的缩放定律：Qwen3.5 监督模型随规模提升，自预测例外

This post is the third in our series on our ongoing efforts to train oversight foundation models: capable AI assistants that can help us to understand other AI models. Our previous write-ups on exact string elicitation and activation oracles and more updates like this can be found on the Oversight Foundations Blog.

Abstract. Oversight models should be able to predict how finetuning changes a subject model's behavior, even when the change is surprising given the finetuning dataset. We train oversight models of varying sizes to predict a finetuned subject model's responses for prompts, given the finetuning dataset. We find power laws for the smooth performance improvements with increasing amounts of training data and model size. The exception is self-prediction: an oversight model that matches the subject model outperforms larger ones. We also present evidence that prediction performance generalizes to harder, out-of-distribution datasets.

Introduction

Finetuning LLMs can lead to unexpected misaligned behaviors. These behaviors can extend far beyond the narrow task a model was finetuned on [1], and can even emerge from datasets that seem benign to language model judges, as we found when mass-producing such datasets. A strong oversight foundation model should be able to predict finetuning effects like these.

As part of our plan for de-risking oversight foundation models, we studied the task of predicting finetuning effects from datasets. The oversight model is given a finetuning dataset, a prompt, and the base subject model's response to that prompt. It should return the response of the subject model after finetuning on the dataset. Concretely, we trained Qwen3.5 models of varying sizes to predict the responses of finetuned Qwen3.5-2B models.

Similar to exact string elicitation, we find that performance follows a power law as we scale model and data size, and that out-of-distribution loss decreases as we decrease the validation loss. However, unlike exact string elicitation, we observe two differences:

• A privileged access effect: the oversight model trained from the same base model as the subject model outperformed larger models and was the best overall.
• An apparent performance floor: even with infinite compute, the model asymptotes at non-zero loss.

The first effect matches previous work on self-introspection [2]. The second is surprising and merits further study. We speculate that the floor reflects the subject model's own response variance.

Method

The oversight model O's goal is to accomplish the following task: given a bundle of documents, Di, and a question in natural language, qij, predict aij, the response of the finetuned subject model Mi to qij sampled at T=0.

Models. Across all experiments, the subject model M is the Qwen3.5-2B base model. We trained oversight models from Qwen3.5 base models of sizes 0.8B, 2B, 4B, and 9B for our main experiments. For additional experiments, we also trained oversight models from a family that does not include the subject model: Qwen3 base models of sizes 0.6B, 1.7B, 4B, 8B, and 14B. About 5% of our training examples exceed Qwen3 models' smaller 32k-token context window; as a result, Qwen3's losses may be slightly inflated. We omitted the examples exceeding the context window length from Qwen3's validation loss calculations.

Training data. We trained the oversight model O on up to 328k examples (Di,qij,aij), generated by a pipeline adapted from De Schamphelaere et al. [3] (Figure 1). To build the bundles, we downloaded 4M documents from FineFineWeb and computed their BGE embeddings [4]. Each bundle Di consists of the k nearest documents to a random seed document by cosine similarity, with k drawn log-uniformly from 8 to 256. No document appears in more than one bundle. For each bundle, we obtained Mi by finetuning the subject model M on Di with low-rank adaptation (LoRA): 3 epochs of AdamW at global batch size 2 and learning rate 2×10−4. We also prompted Claude Haiku 4.5 with the full text of Di to write 10 distinct questions qi0,…,qi9 about its contents. Finally, we prompted Mi with each question qij to obtain its answer aij. Figure 2 shows a few example question-answer pairs.

Training details. We trained the oversight model O for 1 epoch at global batch size 64. For each size, we set learning rate to 2×10−4 or 4×10−4 based on a prior sweep for the best validation loss on 20k train/val examples. The training objective was to minimize the loss, L=NLLO(aij∣Di,qij,aijpre), where aijpre is base subject model M's answer to qij sampled at T=0. In addition to Di, we conditioned the generation on aijpre. This focuses training on the difference arising from finetuning, rather than on learning to talk like the base subject model. O could quickly learn to copy aijpre from only a few examples, and learn to shift its response from aijpre to aij based on Di from the remaining examples.

Evaluations. We evaluated the oversight model O on in-distribution and out-of-distribution (OOD) examples. The in-distribution validation examples were held out from the same pool of examples generated for the training data. The OOD evaluation examples come from our pipeline for discovering datasets with surprising side effects. The pipeline searches for finetuning datasets that read like ordinary post-training data but instill a target behavior. For each of eight behaviors, such as recommending drunk driving, we took the 100 candidate datasets that elicited the behavior most strongly. We finetuned the subject model on each and recorded its responses to prompts that probe the behavior. We scored 1000 of these examples per behavior, drawn from 11 to 20 of the datasets. To account for differences in the behaviors' difficulty, we computed a normalized prediction gap for each behavior, (NLL‾O−NLL‾Mi)/(NLL‾M−NLL‾Mi), where each NLL‾ is the per-token NLL of the finetuned answers aij averaged over the behavior's examples. A gap of 0 means the oversight model predicts the answers as well as the finetuned subject model itself, and 1 means it does no better than the base subject model, so an oversight model achieving <1 performs better than the base subject model. Figure 5 instead plots the raw NLL, which has the same units as the validation loss.

Results

Scaling laws. Performance improved with increasing amounts of training data and model size. Nevertheless, the improvement in oversight model's performance plateaued quickly. We saw this in the similar performance between oversight models based on Qwen3.5 models with 4B and 9B parameters, and between Qwen3 models with 4B, 8B, and 14B parameters.

Based on these observations, we fitted a Chinchilla-style scaling law [5] to the loss curves, i.e. L(N,D)=E+AN−α+BD−β, where E is the irreducible loss floor. D counts the answer tokens the oversight model is trained on; since each example's prompt also includes its document bundle, training reads about 45 tokens per answer token (Table 1). We omitted the Qwen3.5-2B runs from the fits since they are outliers due to the privileged access effect (see below). For Qwen3.5, we obtain E = 0.453, α = 1.60, and β = 0.672, and the compute-optimal model size and data for a training compute budget C grow as N∗∝C0.296 and D∗∝C0.704. Notably, the optimal amount of training data grows faster than the optimal model size. Table 1 shows the model size, data, and cost needed to achieve a loss within a given percentage of E.

Privileged access. The oversight models trained from Qwen3.5-2B, which matches the subject model, outperformed larger models across training, validation, and out-of-distribution evaluations. They are clear outliers when we plot loss against model size (Figure 4). In contrast, the Qwen3 family scales smoothly from 0.6B to 14B. This suggests a similar trend could explain the other Qwen3.5 models.

Generalization. With the largest training set (328k examples), the Qwen3.5 oversight models with 2B and 9B parameters beat the pre-finetuning 2B baseline. Their normalized prediction gaps are 0.58 and 0.86 when averaged over the eight behaviors. The 4B model roughly matched the baseline, with a gap of 0.99. The 0.8B model did not beat it, with a gap of 1.48. The 4B and 9B models had gaps above 1 on two behaviors: 19th century persona and threatening the user. This is likely because these two behaviors have the smallest NLL difference between the base and finetuned subject models (0.19 and 0.35 nats per token). Across base models, the average and individual validation and OOD losses fell with increasing amount of training data (Figure 5). Assuming a power law relationship between OOD loss and validation loss, OOD loss falls slower than validation loss. The 2B model's validation loss fell below the irreducible loss floor due to the privileged access effect. Interestingly, its OOD loss was higher than the power law relationship predicts.

Inadvertent side effects from finetuning have been widely studied in prior works; in finetuning attacks, it was found that finetuning on benign datasets can degrade model safety [6], and that broader misalignment can emerge from finetuning on narrow misaligned behavior [1]. This emergent misalignment can even be transmitted via semantically unrelated data when distilling from a misaligned teacher model in a phenomenon termed subliminal learning [7].

Researchers have invested substantial effort into developing interpretability techniques to identify anomalous model behaviors. One line of work targeted the activation space, such as by training activation oracles to answer questions about their own activations [8, 9]; we previously wrote about our work on scaling activation oracles to trillion-parameter models [10]. To circumvent limitations on the availability of anomaly data for the activation oracle approach, another line of work attempted to analyze weights directly [11].

Recent work trained LoRAcles for interpreting LoRA adapters to detect subtle model behaviors even on out-of-distribution evaluations [3]. Inspired by earlier success in predicting in-distribution model outputs from training data [12] and in tracing an output's diffuse influence across a dataset's examples [13], we trained LoRAcle-like models that interpret finetuning datasets directly instead of the LoRA weights.

In our experiments, we found that the Qwen3.5-2B model outperformed the 4B and 9B models on the prediction task. This was consistent with the privileged access effect described in previous works. Some studies found that models can predict their own completions [14] and their own behavior better than other models do on in-distribution evaluations [15]. However, other studies failed to find introspection on some tasks [16], and the mixed evidence suggests the capability depends on task and context [17]. Since our task involves token prediction, an apparent privileged access effect is perhaps less surprising.

Conclusion

Models can learn to predict finetuning effects from the finetuning dataset. Apart from self-prediction, performance improved smoothly with oversight model size and training data following the scaling laws we fit. When the oversight model was trained from the same base model as the subject model, it outperformed all other models, including larger ones. This is consistent with earlier work on privileged access. Performance improvements generalized to out-of-distribution tasks, though out-of-distribution loss decreased slower than validation loss with increasing training data.

Acknowledgments

We thank Dami Choi, Cassidy Laidlaw, Anish Mudide, and Ziqian Zhong for helpful discussions and feedback on the draft.

Author Contributions

Yue Chen Li led the project, ran the experiments, and wrote the initial draft. Yaowen Ye contributed to the OOD evaluation, compute estimates, writing, and figures. Jacob Steinhardt advised and supervised the project.

Appendix

Fitting scaling law parameters. To assess how well our data constrains the fitted scaling law, we profiled the objective over a grid of possible initializations, pinning subsets of parameters and refitting the rest. We found the irreducible error E to be tightly-determined, while the exponents α and β are less well-constrained.