Baseten Base Labs 分析 on-policy 自蒸馏（OPSD）的偏差问题

Overview

We will start this research update by saying that on-policy self-distillation (OPSD) may remain useful in certain contexts, but we do not believe spending further research efforts on it (or related variants) will be worth it. Reading through the lens of the literature and our own work, we explain why.

What is OPSD?

In distillation overall, the goal is to get a student model q to learn behavior from a teacher model p. Generally we denote such types of post-training methods distillation (as our goal is to distill knowledge from a different “teacher” model). One common form of off-policy distillation samples from the teacher model and trains the student to match the teacher’s distributions along those rollouts:

LD(θ)=Ex∼D, y∼p(⋅∣x)[∑t=1∣y∣DKL(p(⋅∣x,y<t) ∥ qθ(⋅∣x,y<t))].

However, this can create a mismatch between the rollouts the student sees during training and those it generates at inference. For example, if the student tends to make an early mistake the teacher rarely makes, this form of distillation will struggle to fix the mistake.

We then aim to bring more of this training process towards the student: supervise on rollouts generated by the student itself, and restricting teacher supervision in a way that a student can more effectively learn from. This will be how we progress to OPSD.

On-policy distillation (OPD).

Rather than generating the rollouts for distillation from the teacher model, we can generate rollouts from the student model instead:

LOPD(θ)=Ex∼D, y∼qθ(⋅∣x)[∑t=1∣y∣DKL(p(⋅∣x,y<t) ∥ qθ(⋅∣x,y<t))].

This solves one problem in that training rollouts now match what we will likely see at inference time, but this can still bias trained behavior towards the teacher’s preferred continuations, without ensuring improvement on the student’s own ability to solve the task.

On-policy self distillation (OPSD).

Finally, we can go one step further and have the teacher be the same model as q, just with some additional context c unavailable to the student; for example, a ground-truth answer or worked-out solution:

LOPSD(θ)=E(x,c)∼D, y∼qθ(⋅∣x)[∑t=1∣y∣DKL(qθˉ(⋅∣x,c,y<t) ∥ qθ(⋅∣x,y<t))],

where θˉ denotes stopgrad (we don’t pass gradients through, but this is still updated with the student model every step). The hope is that a context-aware student can provide a helpful teacher because it (a) shares the actual student parameters and architecture, and (b) has expert information ingested through context that can guide the student to learn.

There are many variants of OPSD that have been described since the publication of the original 3 papers that began this craze in January and were focused on maths [1], continual learning [2] and code [3]. At their core, these papers offer a claim on density. In typical RLVR, we will receive one scalar reward per potentially long rollout, which in the case of binary rewards is one bit of feedback. Distillation is far more dense, providing up to log⁡∣V∣ bits per token for a vocabulary of size ∣V∣ [4] through access to the teacher’s next-token distribution at each position. In OPSD, the hope is to receive this same per-token supervision with simply the same model and additional context.

Does this actually matter? Beren Millidge has argued that the ‘amount’ of supervision is relatively unimportant compared to where it is applied [5]. For example, the reward of a verifier has very little information but all that it does contain concerns whether the answer is correct. However, in a token-level loss such as OPD or OPSD, many bits – as we shall soon discover – contain minor token choices that are often irrelevant.

This argument does indeed miss that as on-policy updates stay close to the base model they forget less [6, 2]. And yet even this remains disputed. OPSD may retain prior knowledge but also learn little new knowledge [7].

And yet these narrow claims miss the important assumption. That the teacher with some “context” actually points the student in the right direction! Our teacher model already knows the answer and therefore doesn’t have to check, backtrack or even hedge answers. Beren Millidge observes that this dense loss (much like process rewards) thereby mostly add bias, and that this bias especially with heavily trained models, is worse than having more variance [5]. If we believe that a hinted teacher model such as in OPSD would behave the same way, then in this method we are swapping a noisy but well-aimed reward for a clean but biased reward. That is a bad trade.

What does the literature say?

If the hinted teacher points the student somewhere other than the answer, we should see it in the literature. And we do.

One of the first papers that came out demonstrated that on math tasks, OPSD on Qwen3-8B could match and exceed vanilla GRPO after 100 updates with relatively short 1024 token rollouts [1]. This was similar for Self-Distillation Policy Optimization (SDPO) on code [3].

After this initial burst, different training regimes and held-out problems changed these gains. As will be mentioned shortly, having thinking on for OPSD lowers the average accuracy for all five thinking models compared with no thinking [8]. Similarly, testing robustly on held-out coding problems saw SDPO lost 2.5 points on Qwen3-8B, and 9.9 on Olmo-3-7B-Instruct compared to robust gains for GRPO [9].

What does this look like on our own model?

What would a good teacher do? One argument may say that it would be the student model conditioned on “success”. In this scenario the teacher would move each token to the extent that the specific token changes the student's chance of correctly answering the question:

p⋆(yt∣x,y<t)=q(yt∣x,y<t)Vq(x,y≤t)Vq(x,y<t).

Here Vq(⋅) is the probability that the student eventually answers correctly from a given prefix.

Unfortunately this does not match what the teacher model in OPSD does. Indeed, one study has shown that OPSD guides the student model toward simply the teacher model averaged over hints [24].

To better visualize the difference between this ‘average teacher’ and ‘good teacher’, we can sample Qwen3-1.7B with thinking on for 200 hard MATH-500 problems. We can then score every token with log⁡q−log⁡p. Here the teacher is simply the same model shown the reference solution. Note that there is no training involved here.

Figure 1 shows part of a rollout in which the student has already worked out the correct answer and is now double-checking it. But the teacher does not disagree about the maths at all! Rather, it disagrees with irrelevant wording details: “But”, “Wait” and the question mark in “So the answer is 4/9?”. Even more funnily, the largest disagreement occurs when the student model writes 24 − 2*3 rather than the teacher model that writes 24 − 6.

This is also systematic. Across all 800 rollouts (Figure 2), the teacher model consistently votes down semantic or irrelevant tokens such as “Hmm” or “verify”. Swapping the additional context provided to the teacher further affects things. For instance “hedging” words lose roughly half their difference with a different worked solution. These findings are not new observations. Previous work has shown that self-distillation will suppress words such as “wait”, with hints lowering the probability that the teacher places on such hedging tokens such as “Wait” and “Maybe” [10, 11]. This tells us that the direction of this supposedly dense and thereby useful signal can be slightly awry.

This also fits the literature in an interesting way. Two studies with Qwen3-8B (thinking off) have found that using another maths problem’s solution works at least as well as training with the correct one, and on several maths benchmarks better [13, 14]. In part therefore, this suggests the useful part of OPSD comes from behaviour that a related worked solution induces, with the gain being the structure of the answer rather than the answer itself.

Does this phenomenon get worse at scale?

If OPSD does make the trade between variance and bias, then the argument is that this likely to matter more as models scale and get stronger, as the biased signal will no longer move the model to where the true objective (e.g. answering correctly). A simple intuition here is that a stronger model will already reason well, so nudging it with some additional bias such as more hedging will actually make its performance worse.

This thesis is reflected in the findings of the field. Indeed, OPSD helps most where the student would benefit the most from improved reasoning, such as having a non-thinking student model paired with the same model with thinking on [25]. Likewise, student models that reason well already are hurt by OPSD, reducing average accuracy below the base model for five separate thinking models [8]. This is also reflected in the size of the base model, with the beneficial effect of OPSD shrinking proportional to model size, from 3.05 points at Qwen3-1.7B to 0.14 at 8B [21].

Can we correct the teacher’s bias?

Perhaps a natural train of thought after observing a biased teacher is to try and remove the bias.

Removing such bias led to a method we called “contrastive OPSD”, described below. We note the invention of this method was a failed attempt at creating a genuinely useful OPSD variant. However it is important to understand why that itself is a significant finding.

Our reasoning is if the bias of the teacher comes from simply having a source (as above), then comparing and subtracting two different versions of the teacher would leave only the ‘knowledge’ itself. Here we use a factual acquisition experiment mirroring the original SDFT paper: take a corpus of wikipedia articles with facts that are poorly answered by the base model and asks, via OPSD, the model to learn from them.

Contrastive OPSD works with three copies of the teacher, that each score the prefix from the student with three different versions of additional context. First, the anchor policy a sees the question, the wikipedia article and the correct answer. The positive teacher p+ sees the question and the article. The null teacher p− sees the question and the article with the factual sentence removed. Each token is then reweighted by how much more likely the fact makes it:

dt(y)=log⁡pt+(y)−log⁡pt−(y),ut(y)=dt(y)−Ey′∼at[dt(y′)],p~t(y)∝at(y)exp⁡ ⁣(12ut(y)),

where all policies a,p+,p− are evaluated over the same prefix (x,y<t) and each teacher’s additional context. We also clip u to ±10.

The implication here is that tokens that only appear because the teacher has a source with respect to them should cancel out, whilst tokens that explicitly depend on the fact will receive a boost. The student is then trained towards q exactly as in plain OPSD with reverse KL.

Comparing contrastive OPSD (c-OPSD) with OPSD and SFT on Qwen3-4B-Instruct across 400 facts from earthquake and storm wikipedia articles and tested on 800 reworded questions, we can characterize whether such bias can be fixed. Across 10 seeds, all methods climbed initially up to 400 steps. At this stage, when asked what the “max 1-minute wind for storm GREG (2023)” was, all methods answered “45”. However, as you trained c-OPSD and OPSD for longer, the student began to collapse, grading the “reference answer” or “final evidence” and hiding the correct number in a volume of hedging text.

Much of the follow-up work in addition to ours has tried to fix this bias. We believe any application of these fixes without the correct intentions is inherently flawed. We summarize them briefly below:

• Mask and clipping literature. Clipping each token's contribution after finding that style tokens such as “wait” and “hmm” carried several times the KL of maths tokens [1]. RMSD trains only on up to five positions that an LLM judge picks from the 20 biggest disagreements [16].

• Target modification. Methods here remove the reference-induced part of the teacher [17], distill towards a correct context and away from an incorrect one [18], or average the teacher over partially masked references [19]. This is where our contrastive OPSD sits.

• Removing the teacher bias direction. This attempts to remove the bias given by the teacher with respect to the direction of the introduced bias. For example, OASIS distills only along the student’s shortest verified rollout [21]. Cursor adds hints on top of RL [22] and Le Critique gives the hint to a value function [23].

Regardless of the method chosen to remove bias from the teacher, we cannot ensure that the remaining signal promotes the ideal reasoning, rather than shortcutting to the intended answer or verbiage.

Conclusion

It’s quite frequent in this field for ideas and algorithms to get overhyped as universal solutions; not necessarily to the fault of individual contributors, but rather a by-product of a scientific field with an incredibly large audience and publication velocity. OPSD, while originally popularly championed as a solution to continual learning through true on-policy distillation, still exhibits the biased behavior we observe from standard OPD, even through our corrections to the teacher’s bias. OPSD’s use is then more specific than a catch-all RL algorithm: namely, when distilling full contextual behavior that’s expected to be used more universally, such as a system prompt or constitution.

References

1. S. Zhao et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. arXiv:2601.18734, 2026.

2. I. Shenfeld et al. Self-Distillation Enables Continual Learning. arXiv:2601.19897, 2026.

3. J. Hübotter et al. Reinforcement Learning via Self-Distillation. arXiv:2601.20802, 2026.

4. K. Lu and Thinking Machines Lab. On-Policy Distillation. 2025. thinkingmachines.ai/blog/on-policy-distillation

5. B. Millidge. How can LLM RL Work Despite Information-Theoretic Inefficiency. 26 July 2026. beren.io

6. I. Shenfeld, J. Pari and P. Agrawal. RL's Razor: Why Online Reinforcement Learning Forgets Less. arXiv:2509.04259, 2025.

7. C. H. Wu et al. Off-Policy Merging Beats On-Policy Self-Distillation for Continual Learning. arXiv:2610.05872, 2026.

8. S. Kaur et al. Rethinking On-Policy Self-Distillation for Thinking Models. arXiv:2607.05184, 2026.

9. J. K. Adhinarta et al. Does self-distillation improve reasoning? arXiv:2610.18613, 2026.

10. J. Kim et al. Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs? arXiv:2603.24472, 2026.

11. G. Shen et al. Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information. arXiv:2605.11609, 2026.

12. K. Peng et al. Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation. arXiv:2607.10805, 2026.

13. S. Shrestha and A. Tessier. Rethinking Privileged Information in On-Policy Self-Distillation. arXiv:2608.18271, 2026.

14. Y. Ichihara et al. Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation. arXiv:2608.09228, 2026.

15. X.-P. Nguyen et al. Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation. arXiv:2608.09263, 2026.

16. Feng and Vaid. Relevance-Masked Self-Distillation. Applied Compute, 2026. appliedcompute.com

17. Z. Shen et al. Purified OPSD: On-Policy Self-Distillation Without Losing How to Think. arXiv:2607.02234, 2026.

18. A. Baumann et al. On Repulsive and Attractive Teachers: Separating Correctness from Behavior in Self-Distillation. arXiv:2609.21561, 2026.

19. K. Saadi and D. Wang. One Teacher, Many Views: Reducing Reference Sensitivity in On-Policy Self-Distillation. arXiv:2610.30880, 2026.

20. J. Kim et al. Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR. arXiv:2605.10781, 2026.

21. M. I. Hossain et al. Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning. arXiv:2609.37915, 2026.

22. Cursor. Introducing Composer 2.5. 2026. cursor.com/blog/composer-2-5

23. S. Venkatraman, M. Dinot, L. Aitchison. Le Critique: Privileged Value Functions for LLM Reinforcement Learning. arXiv:2608.16739, 2026.

24. C. Yang et al. Self-Distilled RLVR. arXiv:2604.03128, 2026.

25. Y. Li et al. Diagnosing On-Policy Self-Distillation for Reasoning Language Models. arXiv:2609.39118, 2026.