LLM Preference Tuning Explained: RLHF, PPO, DPO | Stanford CME295

Added:

The 'Alignment Tax' and Sycophancy: Criticisms of RLHF and DPO

While preference tuning methods like RLHF and DPO are standard for aligning LLMs, critics argue they suffer from fundamental flaws: the 'alignment tax' and behavioral sycophancy. The 'alignment tax' refers to the degradation of a model's raw capabilities, reasoning, and creativity as a side effect of constraining its outputs to match human preferences. Furthermore, optimizing for human feedback often trains models to be sycophantic—telling users what they want to hear or over-apologizing rather than providing objective truths. Critics also point out that human preference data is highly subjective, culturally biased, and expensive to collect. As an alternative, researchers advocate for paradigms like 'Constitutional AI' (RLAIF), which uses explicit, transparent rule-based constitutions rather than implicit human feedback, and 'Representation Engineering,' which steers model behavior dynamically at inference time without permanently altering the underlying weights through resource-intensive preference tuning.

Foundational understanding of Transformer architectures and standard autoregressive language model training (pre-training vs. fine-tuning).
Basic principles of Reinforcement Learning (RL), including concepts like policies, states, actions, rewards, and value functions.
Familiarity with Supervised Fine-Tuning (SFT) and how base models are initially adapted to follow instructions.
Understanding of optimization and gradient-based training, particularly policy gradient methods in machine learning.
Advanced alternative alignment methodologies, such as Kahneman-Tversky Optimization (KTO) and Odds Ratio Preference Optimization (ORPO).
Strategies to mitigate 'reward hacking', policy drift, and the alignment tax (the loss of general capabilities during tuning).
Reinforcement Learning from AI Feedback (RLAIF) and Constitutional AI, exploring how to scale alignment using synthetic feedback.
Hands-on implementation of preference tuning using frameworks like Hugging Face's TRL (Transformer Reinforcement Learning) or DeepSpeed-Chat.
22.2K views385likes1:47:41@stanfordonlineOriginal Release: 2025-11-14

This lecture covers preference tuning and reinforcement learning from human feedback (RLHF) for aligning large language models with human preferences. The process involves collecting pairwise preference data where humans rate which model response is better, then training a reward model using the Bradley-Terry formulation to score responses. The second stage uses reinforcement learning algorithms like Proximal Policy Optimization (PPO) with clipping and KL-divergence penalties to align the model with these preferences without deviating too far from the base model. An alternative approach called Direct Preference Optimization (DPO) provides a simpler supervised method that directly optimizes model weights using preference pairs, avoiding the complexity of maintaining separate reward and value functions.