Theory

What is GRPO?

Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm designed to train large language models (LLMs) for complex tasks like solving math problems or writing code. Unlike older methods, GRPO is memory-efficient because it doesn't use a separate "value function" (a model that estimates future rewards). Instead, it generates multiple answers for each question, scores them with a reward model, and uses the average score as a reference to decide which answers are better. This makes it easier to train large models on limited hardware, which is surprising because it still performs well on tough tasks like reasoning.

How Does GRPO Work?

GRPO works in simple steps:

For each question, the model generates several possible answers.
Each answer is scored using a reward model (e.g., giving a high score for a correct math answer).
The average score of all answers for that question is calculated.
The model compares each answer's score to this average to see how good it is (this is called the "advantage").
The model then updates itself to favor answers with higher advantages, ensuring it doesn't change too much at once to stay stable.

This process repeats, helping the model get better over time. A surprising detail is how it uses the group average as a baseline, which reduces the need for extra memory while still improving performance.

Why is GRPO Important?

GRPO is important because it saves memory and computational resources, making it easier to train large models on devices with limited power. It's been used in models like DeepSeek R1, which competes with top AI models in reasoning tasks, showing big improvements in math and coding benchmarks.

A Comprehensive Analysis of Group Relative Policy Optimization (GRPO)

Introduction to Reinforcement Learning and Policy Optimization

Reinforcement learning (RL) is a branch of machine learning where an agent learns to make decisions by interacting with an environment, aiming to maximize a cumulative reward. In the context of large language models (LLMs), RL is used to fine-tune these models to align with human preferences and enhance their performance on specific tasks, such as mathematical reasoning or code generation.

Policy optimization is a class of RL algorithms that directly optimize the policy, which is the strategy the agent uses to decide actions based on states. One of the most popular policy optimization algorithms is Proximal Policy Optimization (PPO), known for its stability and efficiency. PPO uses a clipped surrogate objective to prevent large policy updates and relies on a value function to estimate advantages, ensuring stable training.

However, as LLMs grow larger and tasks become more complex, PPO faces challenges, including high memory overhead from maintaining a value function and increased computational costs. The value function, typically another neural network of comparable size to the policy model, estimates the expected future reward from a given state, adding significant resource demands.

To address these limitations, Group Relative Policy Optimization (GRPO) was introduced, first detailed in the DeepSeekMath paper (DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models). GRPO is designed to enhance the reasoning capabilities of LLMs, particularly for mathematical and coding tasks, by eliminating the need for a value function and leveraging group-based advantage estimation.

The Emergence of Group Relative Policy Optimization (GRPO)

GRPO addresses PPO's limitations by introducing a novel reinforcement learning algorithm that simplifies advantage estimation and reduces memory usage. The key innovation lies in its approach to advantage calculation: instead of relying on a separate value function network, GRPO generates multiple responses for each prompt and uses the mean reward of these responses as the baseline. This group-based method reduces variance in advantage estimates and significantly lowers memory usage, making it suitable for training large models on resource-constrained hardware.

GRPO was first applied in the training of DeepSeek R1, an open-source model challenging OpenAI’s o1 in advanced reasoning, as noted in various analyses (DeepSeek R1: Understanding GRPO and Multi-Stage Training | by BavalpreetSinghh | Jan, 2025 | Artificial Intelligence in Plain English). Its effectiveness in improving performance on benchmarks like GSM8K and MATH highlights its potential to revolutionize LLM training for reasoning tasks.

Mathematical Formulation of GRPO

To understand GRPO's mechanics, consider the following formulation, as detailed in resources like The Math Behind DeepSeek: A Deep Dive into Group Relative Policy Optimization (GRPO) | by Sahin Ahmed, Data Scientist | Jan, 2025 | Medium:

For each prompt , generate responses , where . s_j K_j a_{jk} k = 1, 2, ..., K_j
Each response is scored using a reward model, yielding a reward . a_{jk} R_{jk}
Calculate the mean reward for the group, . \bar{R}_j = \frac{1}{K_j} \sum_{k=1}^{K_j} R_{jk}
The advantage for each response is , reflecting how much better or worse the response is compared to the group average. A_{jk} = R_{jk} - \bar{R}_j

The policy update is guided by the following loss function:

\mathcal{L} = - \sum_{j=1}^M \sum_{k=1}^{K_j} \left( \frac{\pi_{\theta}(a_{jk} | s_j)}{\pi_{\theta_{\text{old}}}(a_{jk} | s_j)} A_{jk} \right) + \beta \sum_{j=1}^M \text{KL}(\pi_{\theta}( \cdot | s_j) || \pi_{\theta_{\text{old}}}( \cdot | s_j))

Here:

M is the number of prompts.
\pi_{\theta} is the new policy parameterized by . \theta
\pi_{\theta_{\text{old}}} is the old policy.
\beta is a coefficient controlling the strength of the KL divergence penalty, ensuring the new policy doesn't deviate too far from the old one for stability.

The importance ratio

\frac{\pi_{\theta}(a_{jk} | s_j)}{\pi_{\theta_{\text{old}}}(a_{jk} | s_j)}

for a sequence

a_{jk}

is computed as the product of the ratios for each token in the sequence, reflecting the policy's probability distribution over the entire response.

Implementation Steps of GRPO

Implementing GRPO involves the following steps, as observed in its application to DeepSeekMath and detailed in A Deep Dive into Group Relative Policy Optimization (GRPO) Method: Enhancing Mathematical Reasoning in Open Language Models - MarkTechPost:

Data Preparation: Collect a batch of prompts, typically in chain-of-thought format for reasoning tasks, such as questions from GSM8K and MATH datasets.
Response Generation: For each prompt, generate multiple responses (e.g., 64 samples per question, as used in DeepSeekMath) using the current policy, with a maximum length of 1024 tokens.
Reward Scoring: Use a reward model to assign rewards to each response. The reward model, initially trained on a base model like DeepSeekMath-Base 7B with a learning rate of 2e-5, evaluates response quality based on accuracy and formatting, as noted in AWS | Community | Deep dive into Group Relative Policy Optimization (GRPO).
Advantage Calculation: For each prompt, calculate the mean reward of its responses and compute the advantage for each response.

\bar{R}_j

A_{jk} = R_{jk} - \bar{R}_j

Policy Update: Update the policy parameters to minimize the loss function, with a learning rate of 1e-6 for the policy model, a KL coefficient of 0.04, and a batch size of 1024. Perform a single update per exploration stage to ensure stability, as seen in the training details of DeepSeek R1 (DeepSeek R1: Understanding GRPO and Multi-Stage Training | by BavalpreetSinghh | Jan, 2025 | Artificial Intelligence in Plain English).

This process is iterative, with GRPO improving the model by leveraging data generated during training, making it an online learning algorithm.

Comparison with Other Policy Optimization Methods

To contextualize GRPO, compare it with other methods, as summarized in the following table based on insights from A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shi and r/ChatGPTPro on Reddit: GRPO (Group Relative Policy Optimization) explanation compared to PPO:

Method	Value Function	Advantage Estimation	Stability Mechanism	Memory Usage
PPO	Yes	Uses value function for baseline	Clipped surrogate objective	High (due to value function)
TRPO	Yes	Uses value function, trust region constraint	Hessian-based trust region	High
REINFORCE	No	No baseline, high variance	None	Low
GRPO	No	Group mean as baseline, reduces variance	KL divergence in loss function	Low

PPO: Relies on a value function for advantage estimation, with a clipped importance ratio to prevent large updates. It is stable but memory-intensive, especially for large models.
TRPO: Uses a trust region to constrain policy updates, ensuring stability but at a higher computational cost due to Hessian calculations.
REINFORCE: A basic policy gradient method without constraints, leading to unstable training and high variance, but with low memory usage.
GRPO: Eliminates the value function, using group-based advantages to reduce variance and memory usage, with KL divergence ensuring stable updates. It is particularly efficient for LLMs, as seen in DeepSeek R1's training.

Case Study: Application in DeepSeek R1

DeepSeek R1, an open-source model challenging OpenAI’s o1 in advanced reasoning, utilized GRPO to achieve remarkable results. Introduced in the DeepSeekMath paper, GRPO was applied to DeepSeekMath-Instruct 7B, using a subset of English instruction tuning data (~144K questions). The training details included:

Learning rate for policy model: 1e-6
KL coefficient: 0.04
Samples per question: 64
Max length: 1024
Batch size: 1024
Single update per exploration stage

Performance improvements were significant, as noted in DeepSeek R1: Understanding GRPO and Multi-Stage Training | by BavalpreetSinghh | Jan, 2025 | Artificial Intelligence in Plain English:

GSM8K: Improved from 82.9% to 88.2%
MATH: Improved from 46.8% to 51.7%
CMATH (out-of-domain): Improved from 84.6% to 88.8%

These results highlight GRPO's effectiveness in enhancing mathematical reasoning while optimizing resource usage, making it a game-changer for training LLMs on complex tasks.

Advantages and Potential Disadvantages

Advantages:

Reduced Memory Usage: By eliminating the value function, GRPO requires less memory, crucial for large models, as discussed in AWS | Community | Deep dive into Group Relative Policy Optimization (GRPO).
Simplified Advantage Estimation: Using group means for baseline makes advantage calculation straightforward and efficient, reducing variance, as noted in The Math Behind DeepSeek: A Deep Dive into Group Relative Policy Optimization (GRPO) | by Sahin Ahmed, Data Scientist | Jan, 2025 | Medium.
Stable Training: The KL divergence constraint ensures controlled and stable policy updates, enhancing training reliability.

Potential Disadvantages:

Variance in Group Rewards: If the group size is small, the mean reward might not be a good estimator, leading to higher variance, as mentioned in community discussions (r/ChatGPTPro on Reddit: GRPO (Group Relative Policy Optimization) explanation compared to PPO).
Dependence on Reward Model: The quality of the reward model is critical, as inaccurate rewards can affect performance, a concern highlighted in A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shi.

Conclusion and Future Directions

GRPO represents a significant advancement in reinforcement learning for large language models, offering a more efficient and effective way to train models for complex reasoning tasks. Its application in DeepSeek R1 demonstrates its potential to push the boundaries of AI reasoning, achieving state-of-the-art performance with reduced resource requirements. Future research may focus on optimizing group-based methods, exploring adaptive group sizes, and extending GRPO to other domains beyond mathematics and coding, as suggested in A Deep Dive into Group Relative Policy Optimization (GRPO) Method: Enhancing Mathematical Reasoning in Open Language Models - MarkTechPost.

This comprehensive analysis provides a detailed understanding of GRPO, from its theoretical foundations to practical implementations, serving as a valuable resource for researchers and practitioners in artificial intelligence.

Key Citations