On-Policy Distillation Works Better Without the Teacher
Purdue researchers show teacher models in on-policy distillation act as noise, and introducing OPSA removes heavy GPU compute overhead.
Stock photo for illustration only, not from the actual event
- Purdue researchers found that teacher supervision in on-policy distillation mostly acts as noise.
- Removing the teacher and using simple negative penalties on low-probability tokens matches or beats performance.
- Introduced On-Policy Self-Adaptation (OPSA) to calculate entropy-adaptive negative advantages from the student.
- Eliminates massive GPU memory and compute overhead required to host and run large teacher models.
On-policy distillation has emerged as a standard recipe for training small reasoning models, where student models generate text rollouts while a larger teacher model scores every token along the path to guide policy gradients with dense feedback instead of sparse outcome-level rewards.
However, a new research paper from Purdue University investigators Yi Ding and Ruqi Zhang examined the exact role of the teacher model during training and discovered that teacher supervision is largely composed of noise.
Surprisingly, scaling up the teacher model did not clean up the signal; instead, larger teacher models exhibited higher token-level noise when scoring reasoning paths. When the researchers ablated the noisy teacher advantages and replaced them with a static negative penalty for low-probability tokens, student models achieved identical or even better benchmark performance.
Stock photo for illustration only, not from the actual event
Deep-dive analyses revealed that the training loop does not transfer complex reasoning steps or heuristics. Instead, the loss function simply penalizes tail tokens and pushes probability mass toward the top candidate tokens.
"The training loop was simply using the loss function to penalize tail tokens and push probability mass toward the top candidates."
Yi Ding and Ruqi Zhang
This finding fundamentally shifts our understanding of model distillation, suggesting that its primary function is acting as an entropy regularizer that prunes tail hallucinations rather than transferring deep teacher knowledge.
To eliminate this unnecessary engineering overhead, the authors introduced On-Policy Self-Adaptation (OPSA). Rather than relying on a secondary scoring model, OPSA computes entropy-adaptive negative advantages directly from the student model's own distribution, applying stronger penalties at high-entropy token branching points.
Dropping the teacher model removes substantial operational hurdles, saving valuable GPU memory and compute cycles that would otherwise be wasted on teacher forward passes for every generated token during training.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment