Simple Reinforcement Learning Implementations

This repository contains minimal, educational implementations of three key reinforcement learning algorithms: DPO (Direct Preference Optimization), PPO (Proximal Policy Optimization), and GRPO (Group Relative Policy Optimization).

Each implementation focuses on clarity and intuition over complexity, using pure Python with minimal dependencies to demonstrate core concepts.

Files Overview

• simple_dpo.py - Direct Preference Optimization for learning from human preferences
• simple_ppo.py - Proximal Policy Optimization with actor-critic architecture
• simple_grpo.py - Group Relative Policy Optimization using relative comparisons

---

Section 1: Core Concepts & Algorithms

1. DPO learns preferences directly without reward modeling

Unlike traditional RLHF which requires training a separate reward model, DPO optimizes preferences directly using paired comparison data. The algorithm uses sigmoid probability to convert score differences into preference likelihoods: prob = sigmoid(� � (preferred_score - rejected_score)).

2. PPO balances learning with stability through clipping

PPO prevents destructive policy updates by clipping the probability ratio between old and new policies. The clipped ratio max(1-�, min(1+�, ratio)) ensures gradual learning while maintaining training stability, solving the exploration-exploitation balance.

3. GRPO eliminates the need for value networks through group comparison

Instead of training a separate critic to estimate state values, GRPO calculates advantages by comparing responses within each generated group. This reduces computational requirements by ~50% while maintaining learning effectiveness through relative ranking.

4. Advantage functions drive policy improvement in all three algorithms

• DPO: advantage = (prob - 1) measures prediction confidence error
• PPO: advantage = reward - value_estimate compares actual vs expected outcomes
• GRPO: advantage = (reward - group_mean) / group_std normalizes performance within group context

5. Gradient ascent vs descent depends on optimization target

DPO and PPO use gradient ascent because they maximize expected reward/preference probability, while traditional ML minimizes loss. The key insight: new_param = old_param + learning_rate � gradient for maximization problems.

---

Section 2: Implementation Details & Code Patterns

1. Probability sampling follows the cumulative distribution pattern

All three implementations use the same cumulative probability sampling technique:

rand = random.random()
cumulative = 0
for i, prob in enumerate(probabilities):
    cumulative += prob
    if rand <= cumulative:
        selected_index = i
        break

This creates a "roulette wheel" where each option gets space proportional to its probability.

2. Policy updates modify probabilities of actually chosen actions

The implementations correctly track action_idx/answer_type_idx to ensure updates target the sampled action rather than always updating the first option. This prevents systematic bias toward specific actions and enables proper learning from experience.

3. Negative log-likelihood loss punishes wrong predictions exponentially

DPO uses loss = -log(prob + 1e-8) where the loss increases dramatically as probability approaches 0. This creates strong learning signals: confident correct predictions have tiny loss (~0.05), while wrong predictions have large loss (~2.3).

4. Epsilon constants prevent numerical instabilities

The + 1e-8 terms throughout the code prevent log(0) and division by zero errors. These tiny values (0.00000001) don't affect meaningful computations but ensure mathematical stability during edge cases.

5. Probability normalization maintains valid distributions

After each update, probabilities are normalized to sum to 1:

total = sum(policy_probs)
policy_probs = [p / total for p in policy_probs]

This ensures the policy remains a valid probability distribution despite individual probability modifications.

---

Section 3: Practical Insights & Comparisons

1. Training data requirements differ significantly across methods

• DPO: Requires preference pairs (A > B) from human annotations
• PPO: Needs environmental rewards for individual actions/states
• GRPO: Uses group-generated responses with quality scores Each approach suits different scenarios: DPO for human alignment, PPO for interactive environments, GRPO for response quality improvement.

2. Computational complexity varies by architectural choices

GRPO achieves ~50% compute reduction by eliminating the value network, while PPO requires both actor and critic updates. DPO has the simplest architecture (single weight) but needs careful preference data curation. The trade-off: complexity vs. generality.

3. Learning stability mechanisms serve different purposes

• PPO clipping: Prevents catastrophic policy changes via ratio constraints
• KL divergence penalty: Keeps new policy close to old policy
• Group normalization: GRPO's stability comes from relative comparisons rather than absolute values Each mechanism addresses the fundamental challenge of stable learning in high-dimensional policy spaces.

4. Convergence behavior reflects algorithm design philosophy

DPO converges when preference probabilities match training data labels. PPO converges when policy maximizes expected rewards. GRPO converges when response quality rankings stabilize within groups. Understanding these end states helps debug training issues.

5. Real-world scaling requires architectural adaptations

These minimal implementations demonstrate core concepts but lack features needed for production use: neural network policies, batch processing, distributed training, and sophisticated exploration strategies. The principles remain the same, but implementation complexity increases dramatically for practical applications.

---

Loss Functions Explained Simply

DPO Loss: How wrong are our preference predictions?

loss = -log(prob + 1e-8)

• What it measures: "How surprised are we that the model gave this probability?"
• When prob = 0.95 (confident, correct): loss = 0.05 (tiny penalty)
• When prob = 0.20 (wrong): loss = 1.61 (big penalty)
• Why it works: Punishes wrong predictions exponentially - being slightly wrong costs a little, being very wrong costs A LOT

PPO Loss: Combined policy and stability costs

policy_loss = -(advantage × log(new_prob)) + kl_penalty × kl_divergence

• Part 1: -(advantage × log(new_prob)) = "How much did this action help/hurt?"
• Part 2: kl_penalty × kl_divergence = "Did we change the policy too much?"
• Total: Learning signal + stability penalty
• Why it works: Balances improvement (make good actions more likely) with caution (don't change too fast)

GRPO Loss: Implicit through direct probability updates

# No explicit loss function - uses direct updates:
new_prob = old_prob + learning_rate × advantage × old_prob

• What it measures: No separate loss calculation - updates probabilities directly based on group performance
• How it works: If group performance is above average (positive advantage), increase probability
• Why it's different: Skips loss calculation entirely, making updates based on relative group rankings
• Advantage: Simpler math, but less interpretable than explicit loss functions

Key Insight:

All three algorithms optimize different things:

• DPO: Minimize prediction error on preference data
• PPO: Maximize expected reward while staying stable
• GRPO: Maximize relative performance within groups

---

Key Takeaways

For Understanding RL: These implementations show how modern RL algorithms work at their core - updating probability distributions based on feedback signals to improve decision-making over time.

For Practical Application: Choose DPO for preference learning, PPO for general RL tasks, and GRPO for efficient response generation. Each algorithm solves different problems with different trade-offs.

For Further Learning: The mathematical foundations demonstrated here (sigmoid functions, log-likelihood, policy gradients) appear throughout modern AI research. Mastering these building blocks enables understanding of more complex algorithms.

---

These implementations prioritize educational clarity over performance. For production use, consider established libraries like OpenAI Baselines, Stable-Baselines3, or TRL.