Multi-Reward RL for LLM Agents: PPO, GRPO, DAPO, GDPO
Comparing 4 RL trainer configurations for LLM agents, exploring Scale Dominance issues, and GDPO's Normalize-then-Sum solution.
Stock photo for illustration only, not from the actual event
- Training autonomous agents with a single scalar reward fails due to conflicting real-world priorities.
- GRPO cuts GPU memory needs in half but suffers from Scale Dominance and Dead Groups.
- DAPO solves Dead Groups with dynamic sampling, discarding wasted rollout data immediately.
- GDPO applies Normalize-then-Sum, normalizing each reward channel independently before combination.
New empirical follow-up: Part 2 compares seven trainer configurations on Qwen3-14B and our DEX gym, featuring no-think and thinking holdouts, interactive reward curves, and downloadable data. This is a separate experiment from the 27B table and does not establish a universal trainer ranking.
If you only ever train language models on toy math puzzles, reinforcement learning feels simple: did the model output 42? If yes, reward is 1. If no, reward is 0. But the moment you try to train an autonomous agent for real-world enterprise work, a single scalar reward is an absolute illusion.
In production, your agent has to juggle multiple competing, messy, non-commensurate priorities at the same time:
- Task correctness
- Token efficiency
- Safety and constraint enforcement
Stock photo for illustration only, not from the actual event
Here is the dirty secret of post-training: if you take these three scores and simply add them together into standard algorithms like PPO or vanilla GRPO, your training run will almost certainly tear itself apart. The loudest reward channel swallows the subtle ones, the agent learns to game the system, and up to a third of your expensive GPU batches end up generating zero gradients.
Understanding how several distinct reward channels are combined into policy advantages helps explain why simple weighted sums cause catastrophic scale dominance, how GDPO decouples channel normalization, and why hard constraints require enforcement rather than reward weights.
To understand why modern multi-reward agent post-training looks the way it does, we must trace how policy gradient estimators evolved:
- Proximal Policy Optimization (Schulman et al., 2017) uses an Actor-Critic architecture where the Actor generates tokens and a Critic predicts future rewards.
- DeepSeekMath (2024) introduced Group Relative Policy Optimization (GRPO), eliminating the Critic network entirely.
Instead of using a neural net to predict a baseline, GRPO samples a group of G candidate completions {o₁, o₂, ..., o_G} for the same prompt, scores them all, and normalizes advantages against the group’s own mean and standard deviation.
When you run GRPO on difficult engineering problems, you quickly discover the curse of Dead Groups. If a coding task is tough and all 8 candidate rollouts in a group fail with a syntax error, every single rollout gets a reward of 0, yielding zero group variance and completely empty gradient updates.
"A group where all 8 attempts fail has zero variance and gives no gradient. DAPO's dynamic sampling replaces it with a prompt whose attempts differ."
Dev.to
DAPO introduced dynamic sampling: during rollout scoring, if a group has zero variance, it immediately discards the dead data and pulls fresh active prompts until every training batch contains real learning signal, cutting wasted GPU cycles to near zero.
Introduced in 2026 (arXiv:2601.05242) and integrated into modern libraries like TRL 1.7, GDPO fixes the fundamental multi-reward flaw of GRPO. Instead of adding raw scores together and then normalizing, GDPO enforces Normalize-then-Sum.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment