Academic Databases
Explore Our Semantic Databases
STEM & Tech
Math, Physics, CS & Engineering...
Economics
Macro, Micro, Econometrics...
Business
Management, Marketing, Finance...
Health & Life Sciences
Biology, Medicine, Neuroscience...
Humanities & Social Sciences
Psychology, Law, Philosophy, History...
General Studies
General & interdisciplinary studies...
Explore Topics
Features

Direct Preference Optimization (DPO): Fine-Tune LLMs Without RL

Added: 2026-09-10

Deep Dive

Supervised Fine-Tuning (SFT) principles, including how LLMs are adapted to instruction-following tasks.

How AI Learns to Follow Instructions ? | SFT

How AI Learns to Follow Instructions ? | SFT

136

Supervised Fine-Tuning (SFT) is a training technique that transforms raw language models into helpful assistants by teaching them to follow instructions, using the same algorithm as pre-training but with a loss mask that only grades response tokens, requiring only 1,000-10,000 examples to achieve significant quality improvements, with quality plateauing early at around 85% and demonstrating that knowledge comes from pre-training while behavior comes from SFT.

Click to watch →

Building Large Language Models: Stanford CS229 Lecture

Building Large Language Models: Stanford CS229 Lecture

Supervised Fine-Tuning (SFT) is the post-training technique used to align pre-trained language models into useful AI assistants. The process involves fine-tuning the pre-trained model on human-labeled question-answer pairs, where humans provide desired responses to specific prompts. SFT teaches the model to follow instructions and produce helpful answers rather than just modeling internet text patterns. Despite requiring relatively little data (around 2,000-32,000 examples), SFT dramatically improves practical usefulness by specializing the general-purpose language model toward helpful assistant behavior.

Click to watch →

Finetuning Llama3.2 3B with an IITian | ML/LLM Project

Finetuning Llama3.2 3B with an IITian | ML/LLM Project

9.3K

Supervised Fine-Tuning (SFT) transforms pre-trained LLMs into instruction-following assistants by retraining on instruction-answer datasets. The goal is not to add new knowledge but to adapt models to specific tasks like document generation or PDF explanation. SFT works best when leveraging existing knowledge in the base model; attempting to teach completely new domains leads to hallucinations. Three main techniques exist: Full Fine-Tuning (retraining all parameters, impractical due to resource requirements and catastrophic forgetting), LoRA (Low-Rank Adaptation, freezes weights and adds small adapter matrices for <1% trainable parameters), and QLoRA (Quantization-Aware LoRA, provides additional 33% memory reduction). Unsloth library offers 2x faster training and 60% memory reduction for Llama models. Essential libraries include Unsloth, Transformers, TRL, and PyTorch. Model loading uses FastLanguageModel.from_pretrained with specific model names and parameters like max_sequence_length (2048 tokens) and load_in_4bit=True for memory efficiency.

Click to watch →

Ali Ghodsi, Deep Learning, RLHF, GhatGPT, Alignment in LLMs, Fall 2023, Lecture 14

Ali Ghodsi, Deep Learning, RLHF, GhatGPT, Alignment in LLMs, Fall 2023, Lecture 14

2.2K

SFT creates instruction-response datasets where prompts (questions, requests, tasks) are paired with human-written responses. Prompts are categorized as plain (arbitrary questions), few-shot (with examples), or user-based (generalized API requests). The model concatenates prompts and responses, learning to predict response tokens. This enables the model to handle diverse NLP tasks like brainstorming, classification, extraction, generation, and summarization by casting them as next token prediction problems.

Click to watch →

Fine-tuning LLM: fakty i mity

Fine-tuning LLM: fakty i mity

24.4K

Supervised Fine-tuning (SFT) is the primary fine-tuning approach where models are trained on instruction-response pairs. Humans provide instructions and expected responses, and models learn to generate appropriate responses. This phase enables models to follow specific commands, maintain conversation context, and produce structured outputs. SFT is the bridge between pre-trained models and practical applications, transforming models from text generators into task-oriented assistants.

Click to watch →

Deep Dive

The standard Reinforcement Learning from Human Feedback (RLHF) pipeline, specifically the division between the policy model and the reward model.

CMU Advanced NLP Fall 2025 (16): Reinforcement Learning for LLMs

CMU Advanced NLP Fall 2025 (16): Reinforcement Learning for LLMs

641

RLHF (Reinforcement Learning from Human Feedback) pipeline consists of three steps: (1) Supervised fine-tuning on demonstration data (prompts and responses), (2) Train a reward model using preference data (pairs of outputs where one is preferred over the other), (3) Optimize the policy against the reward model using reinforcement learning. The Bradley-Terry model provides a principled approach: the probability that output i is preferred over j is exp(r_i) / (exp(r_i) + exp(r_j)), leading to a loss function that maximizes likelihood of observed preferences. This pipeline was popularized by OpenAI in 2022 and has become standard for aligning language models with human preferences.

Click to watch →

Lecture 38 : Reinforcement Learning from Human Feedback - I

Lecture 38 : Reinforcement Learning from Human Feedback - I

394

The RLHF pipeline consists of three phases: collecting human feedback, training a reward model, and training the policy model. Unlike pre-training and instruction fine-tuning which use next-token prediction, RLHF employs binary classification. The core challenge is obtaining rewards without human-in-the-loop at runtime. Reward modeling solves this by training a separate model to predict scalar rewards for prompt-completion pairs, mimicking human scoring preferences. Preference data collection uses pairwise comparisons rather than absolute scoring because human annotators vary significantly in leniency. This comparison-based approach reduces annotation variance and provides clearer training signals for the reward model.

Click to watch →

DPO : Direct Preference Optimization

DPO : Direct Preference Optimization

340

The standard RLHF pipeline consists of three stages: (1) Unsupervised pre-training where language models learn token sequence distributions from large text corpora without human annotations, (2) Supervised Fine-Tuning (SFT) where models are trained on high-quality labeled data for specific tasks like question answering or summarization, and (3) RLHF which involves reward modeling and policy optimization. In reward modeling, human labelers annotate preferences between response pairs sampled from the SFT model. The Bradley-Terry model is used, assuming preferences follow a sigmoid function of reward differences: P(y1 > y2 | x) = sigmoid(R(x, y1) - R(x, y2)). A binary classification loss is minimized to train the reward model, which can use the same Transformer architecture as the SFT model with an additional linear layer for reward scores.

Click to watch →

Modern Reinforcement Learning (RL), Part 1: How RL Powers Generative AI

Modern Reinforcement Learning (RL), Part 1: How RL Powers Generative AI

114

RLHF follows a traditional RL pipeline for aligning LLMs with human preference. It consists of three stages: supervised fine-tuning on curated high-quality responses to establish behavioral foundations; reward model training where human annotators compare response pairs to train a model that scores satisfaction and alignment; and policy optimization using algorithms like PPO to maximize learned reward while staying close to original behavior through KL divergence penalty. PPO has become the industry standard due to its stability - it offers controlled optimization preventing aggressive updates that could destabilize training, is robust against noisy human feedback, uses clipping mechanisms to keep updates within safe bounds, and scales well for billion-parameter models.

Click to watch →

What is RLHF?

What is RLHF?

5.7K

RLHF training involves three steps: (1) Collect demonstration data and train a supervised policy using clean prompt-answer pairs, (2) Collect comparison data and train a reward model by having human labelers rank multiple model outputs, (3) Optimize a policy against the reward model using PPO (Proximal Policy Optimization). PPO is a fundamental RL algorithm considered one of the easiest to implement, widely available in open-source libraries. The reward model serves as the value function that predicts rewards for given states (prompts) and actions (model outputs).

Click to watch →

Deep Dive

Basic Reinforcement Learning concepts, such as policy optimization, actor-critic frameworks, and Kullback-Leibler (KL) divergence regularization.

StatQuest with Josh Starmer is live!

StatQuest with Josh Starmer is live!

9K

KL Divergence (Kullback-Leibler Divergence) is a measure of how different one probability distribution is from another. In reinforcement learning, it is used to prevent the model from changing too dramatically from its previous version during training. This regularization term ensures that the new policy is similar enough to the old policy to maintain useful capabilities while still improving. The balance between improvement and stability is crucial for successful reinforcement learning.

Click to watch →

Munchausen Reinforcement Learning

Munchausen Reinforcement Learning

1K

Kullback-Leibler (KL) regularization in RL frames policy updates as penalizing greediness by the KL divergence between the new policy and previous one. By adding entropy to the objective, policies become sufficiently stochastic. The solution to this optimization problem can be computed analytically as a softmax over the sum of all past Q-functions. Setting α=0 gives entropy regularization; setting α=1 gives KL regularization. This framework is reminiscent of algorithms like TRPO and MPO, and both schemes are mathematically equivalent through reparameterization of the Q-function.

Click to watch →

Martha White | Better Actor-Critic Algorithms for RL | June 5, 2025

Martha White | Better Actor-Critic Algorithms for RL | June 5, 2025

253

Actor updates minimize KL divergence to target policies (Boltzmann or KL-penalized). Forward KL is mean-seeking and maintains broad support; reverse KL is mode-seeking and concentrates on specific modes. Research shows improvement guarantees require sufficient KL reduction relative to critic error. These interacting errors between policy evaluation and greedification explain algorithmic fragility. Forward KL is used in TRPO and Mirror Descent; reverse KL appears in SAC. The distinction matters because forward KL prevents premature convergence to local optima, while reverse KL can cause mode-seeking behavior that gets stuck in suboptimal solutions. Understanding these theoretical differences helps explain why actor-critic algorithms are finicky and requires careful balancing of approximation errors.

Click to watch →

16. المحاضرة العاشرة (شرح Taylor Series - Trust Region - Lagrange Multiplier) فى TRPO | بالعربى RL

16. المحاضرة العاشرة (شرح Taylor Series - Trust Region - Lagrange Multiplier) فى TRPO | بالعربى RL

190

Kullback-Leibler (KL) Divergence measures how different one probability distribution is from another. In TRPO, it is used to constrain the difference between the old policy and the new policy: KL_divergence(old_policy, new_policy) ≤ δ, where δ is a small constant. This ensures stable policy updates.

Click to watch →

AVOID This LLM Reasoning Training Pitfall - KL Divergence

AVOID This LLM Reasoning Training Pitfall - KL Divergence

320

Large language models use probability distributions where all token probabilities sum to 100%. KL (Kullback-Leibler) divergence constrains how much new probability distributions can change from old ones during reinforcement learning. Forward KL measures divergence from old to new policy, ensuring new policy assigns nonzero probability to all classes that old policy had, preventing forgetting plausible options. Reverse KL is opposite, concentrating probability on high-probability options from old policy. Forward KL encourages exploration and preserves model diversity, while reverse KL prevents overconfidence in unlikely options. The goal is making new-to-old probability ratio close to 1. Normalized distributions sum to 100%, while unnormalized represent scores. K1 estimates normalized reverse KL, K3 provides exact unnormalized KL values.

Click to watch →

Deep Dive

Foundational deep learning optimization techniques, particularly binary cross-entropy loss and gradient descent.

ATPESC 2022 8.2 Deep Learning Methods | Tanwi Mallick

ATPESC 2022 8.2 Deep Learning Methods | Tanwi Mallick

105

Loss functions measure prediction error: binary cross entropy for classification (heavily penalizes wrong predictions), mean squared error for regression (squared differences between predicted and actual values). Gradient descent minimizes loss by: (1) randomly initializing weights, (2) computing gradients (partial derivatives of loss w.r.t. weights), (3) updating weights in the opposite direction of the gradient, (4) repeating until convergence. The learning rate controls step size - too high causes instability, too low causes slow convergence or local minima. Backpropagation applies the chain rule to compute gradients across all layers.

Click to watch →

Binary Cross-Entropy

Binary Cross-Entropy

14.4K

Binary cross-entropy is a cost function used for binary classification that addresses the vanishing gradient problem inherent in using mean squared error with sigmoid activation functions; unlike MSE which involves the derivative of the sigmoid function (Sigma prime) leading to small gradients when starting far from optimal values, binary cross-entropy eliminates this term through its mathematical formulation C = -[y log(a) + (1-y) log(1-a)], enabling more reliable gradient descent optimization.

Click to watch →

Introduction to Deep Learning (I2DL 2023) - 2. ML Basics

Introduction to Deep Learning (I2DL 2023) - 2. ML Basics

16.2K

For classification, maximum likelihood finds parameters that maximize the probability of observing true labels. The binary cross-entropy loss is: L = -[y*log(y_hat) + (1-y)*log(1-y_hat)]. This loss penalizes confident wrong predictions heavily and is convex, ensuring a unique global minimum. The cost function is the average loss over all training samples. Gradient descent is used to minimize this loss, with the update rule similar to linear regression but with the gradient of cross-entropy loss. This iterative method is essential for training complex models.

Click to watch →

Convolutional Neural Network from Scratch | Mathematics & Python Code

Convolutional Neural Network from Scratch | Mathematics & Python Code

243.3K

Binary cross-entropy loss measures the performance of a binary classifier by comparing predicted probabilities to true labels (0 or 1). The loss function is defined as -[y_true × log(y_pred) + (1 - y_true) × log(1 - y_pred)]. The derivative with respect to predictions is (y_pred - y_true) / (y_pred × (1 - y_pred)), which drives gradient descent optimization for binary classification tasks.

Click to watch →

Gradient Descent -2

Gradient Descent -2

1.6K

The cross-entropy loss function has important properties: it is convex (has a single global minimum), it is differentiable (allows gradient-based optimization), and it provides strong gradients when predictions are wrong. These properties make it particularly suitable for training neural networks and other models using gradient descent. The gradient descent algorithm for logistic regression follows these steps: (1) Initialize W and B to small random values, (2) Compute the gradient ∇J(W) = Σ (h_θ(x^(i)) - y^(i)) x^(i), (3) Update W := W - α ∇J(W), (4) Repeat until convergence. The learning rate α controls how large each update step is.

Click to watch →

DPO Intro

RLHF Recap

Core Loss

Reward Probabilities

Softmax Details

KL Regularization

Model Stability

Loss Function

Loss Intuition

DPO Summary

DPO Intro

0:00

Playing Section

1
Introduces Direct Preference Optimization, a method for fine-tuning LLMs without reinforcement learning.
2
Recaps the previous RLHF method and the need for a more efficient direct approach.
3
Outlines the goal to show how DPO bypasses the separate reward model training.

Deep Dive

Hands-on implementation of DPO using practical frameworks such as Hugging Face's TRL (Transformer Reinforcement Learning) library.

Online DPO Finetuning for LLMs on Custom Data - Hands-on Tutorial

Online DPO Finetuning for LLMs on Custom Data - Hands-on Tutorial

This section demonstrates practical implementation of Online DPO using Hugging Face's TRL library. The process involves: (1) creating a virtual environment and installing prerequisites (torch, transformers, TRL); (2) cloning the TRL repository from the recent branch containing the new feature; (3) launching Jupyter Notebook for the demonstration; (4) defining the model, reference model, and reward model; (5) preparing training and evaluation datasets; (6) running the Online DPO trainer. The training generates statistics including loss reduction and runtime metrics. The resulting fine-tuned model is saved with checkpoints, tokenizer configurations, and vocabulary files, enabling users to create custom-aligned models from their own datasets.

Click to watch →

09: Alignment II & Merging [Session 9 of Full Course, LLM Engineering Cohort 3]

09: Alignment II & Merging [Session 9 of Full Course, LLM Engineering Cohort 3]

149

This comprehensive section covers the complete practical workflow for implementing DPO in real-world scenarios. It details the technical implementation using libraries like TRL, the critical importance of low learning rates to prevent undesirable artifacts, and the distinction between reference and policy models. The section demonstrates a concrete example reducing toxicity in Mistral 7B from 0.03 to 0.015 using only 500 training samples and 250 steps. Key insights include the importance of monitoring multiple evaluation metrics rather than relying solely on final checkpoints, and understanding that the final saved state is rarely the optimal model.

Click to watch →

[인공지능,머신러닝,딥러닝] (심화) Direct preference optimization (DPO)

[인공지능,머신러닝,딥러닝] (심화) Direct preference optimization (DPO)

2.9K

DPO is implemented in Hugging Face TRL with four parameters: preferred responses (a), rejected responses (b), reference policy (c), and model policy (d). The loss is computed as the difference between preferred and rejected policy ratios. Label smoothing (λ) controls preference data influence, handling unreliable human preferences. Three experimental tasks evaluate DPO: (1) Controlled sentiment generation using IMDB dataset with sentiment classifier as ideal reward model, (2) Summarization using Reddit TL;DR dataset with human-labeled preferences, (3) Single-turn dialogue using Alpaca dataset. Each task uses different SFT baselines and evaluation methods, with GPT-4 serving as judge for summarization and dialogue tasks.

Click to watch →

Fine-tuning LLMs on Human Feedback (RLHF + DPO)

Fine-tuning LLMs on Human Feedback (RLHF + DPO)

20.3K

Implementing DPO requires several technical steps: importing necessary libraries (TRL for training, Transformers for model handling, PyTorch for computation), loading the base model and tokenizer, preparing the preference dataset in the required format (prompt, chosen response, rejected response), and configuring training arguments including batch size, number of epochs, evaluation metrics, and checkpoint saving frequency. The training process typically takes around 8 minutes per epoch on consumer hardware. During training, metrics such as training loss, validation loss, reward margins (difference between preferred and dispreferred response scores), and reward accuracy (percentage of correct preference predictions) are monitored to assess model performance.

Click to watch →

DPO Explained: Direct Preference Optimization Math for LLM Alignment

DPO Explained: Direct Preference Optimization Math for LLM Alignment

DPO implementation requires computing log probabilities for both the optimized and reference language models. This involves combining questions and answers into input strings, passing them through Transformer models to obtain logits, applying log_softmax to get log probabilities, and summing only the probabilities corresponding to actual answer tokens. The Hugging Face DPO trainer automates this process, handling the technical details of probability computation while allowing users to specify hyperparameters like beta that control the strength of the KL divergence constraint.

Click to watch →

Deep Dive

Advanced variations of preference optimization, including IPO (Identity Preference Optimization) and KTO (Kahneman-Tversky Optimization).

KKRGENAI - FREE GENERATIVE AI WORKSHOP - 30 HOURS

KKRGENAI - FREE GENERATIVE AI WORKSHOP - 30 HOURS

132

Advanced preference optimization methods include ORPO (Odd Ratio Preference Optimization), which combines supervised fine-tuning and preference optimization into a single objective. IPO (Identity Preference Optimization) addresses overfitting by introducing theoretical constraints that prevent models from becoming too deterministic. CPO (Contrastive Preference Optimization) trains models by contrasting preferred and dispreferred outputs, working in conjunction with other techniques. These methods provide comprehensive validation mechanisms, ensuring models avoid outputs that are accepted but not ideal while remaining non-deterministic and not overly sensitive to specific areas.

Click to watch →

[2024 Best AI Paper] SimPO: Simple Preference Optimization with a Reference-Free Reward

[2024 Best AI Paper] SimPO: Simple Preference Optimization with a Reference-Free Reward

163

This section presents the comprehensive experimental methodology and comparative analysis framework for evaluating preference optimization methods. Three primary evaluation benchmarks are established: Alpaka Val 2 (805 examples, GPT 4 Turbo baseline, pairwise comparison), Arena Hard (500 examples, GPT 4-0314 baseline), and Empty Bench (80 examples, single-answer grading). Various preference optimization methods are systematically compared including RHF, IPO, CPO, KTO, ORPO, and RDPO. The study employs rigorous hyperparameter tuning with learning rates ranging from 3×10^(-7) to 1×10^(-6), batch sizes of 128, and maximum sequence lengths of 2048. Ablation studies reveal that removing length normalization causes significant performance degradation with longer, repetitive outputs, while improper gamma settings similarly harm results. The analysis demonstrates that both design elements are essential for maintaining model effectiveness.

Click to watch →

Aligning LLMs with Direct Preference Optimization

Aligning LLMs with Direct Preference Optimization

33.9K

IPO improves upon DPO by adding regularization to prevent overfitting to chosen responses. DeepMind researchers observed that DPO risks overfitting when the model becomes too confident in predicting chosen responses. By adding a regularization term similar to weight decay in regression, IPO prevents this overfitting while maintaining alignment quality. Experiments show IPO achieves roughly comparable results to DPO with some trade-offs depending on hyperparameters, offering practitioners another tool in the alignment toolkit.

Click to watch →

Human Alignment of Large Language Models throughOnline Preference Optimisation

Human Alignment of Large Language Models throughOnline Preference Optimisation

108

Two new approaches have been proposed to optimize against preference probabilities directly rather than using the Bradley-Terry model. Identity Preference Optimization (IPO) optimizes preference probabilities against a fixed data distribution using an offline contrastive loss. Nash MDPG aims to find a Nash equilibrium with respect to preference probabilities through online policy gradient updates against a regularized policy. These two algorithms have different characteristics—offline versus online data usage and different nature of equilibria—but are both effective in their own ways.

Click to watch →

767: Open-Source LLM Libraries and Techniques — with Dr. Sebastian Raschka

767: Open-Source LLM Libraries and Techniques — with Dr. Sebastian Raschka

3.7K

KTO (KTO Optimization) is another alternative to RLHF that was developed by researchers including Nobel laureate Daniel Kahneman. Unlike DPO which requires pairs of preferred and rejected responses, KTO can work with absolute scores (single scores for each response) rather than pairwise comparisons. This makes KTO more applicable to scenarios where only absolute ratings are available rather than comparative preferences. The method is named after Kahneman and Tversky, famous social scientists and economists known for their work on prospect theory and decision-making under uncertainty. KTO represents an ongoing effort to develop more efficient and accessible methods for aligning LLMs with human preferences.

Click to watch →

Deep Dive

Best practices for curating and filtering high-quality preference datasets (chosen vs. rejected completions).

LLM Fine-Tuning 16: Preference Alignment & Preference Training in LLMs with RLHF, RLAIF, DPO, LoRA

LLM Fine-Tuning 16: Preference Alignment & Preference Training in LLMs with RLHF, RLAIF, DPO, LoRA

1.7K

Preference training datasets contain three main columns: (1) Prompt - the input question or instruction given to the model, (2) Chosen - the response that should be selected/preferred, and (3) Rejected - the response that should be rejected. Examples include datasets from Hugging Face such as 'anthropic/hh_rlhf' and 'ultrafeedback_binarized_clean', which provide ranked responses where human annotators have labeled which response is better.

Click to watch →

How we cleaned OpenBMB UltraFeedback and Notus

How we cleaned OpenBMB UltraFeedback and Notus

225

Creating high-quality preference datasets for LLM alignment requires systematic curation. The process begins with collecting diverse instructions from multiple sources (UltraChat, SharedGBT, Flan), generating responses with multiple models, and annotating them using detailed schemas (instruction following, truthfulness, honesty, helpfulness). Critical quality checks include verifying score calculations (discovering bugs where low scores were multiplied by 10), removing training data leakage examples, filtering incomplete annotations, and eliminating ambiguous cases. This rigorous pipeline transforms raw data into reliable training material, demonstrating that data quality directly impacts model performance.

Click to watch →

Fine-tuning LLMs on Human Feedback (RLHF + DPO)

Fine-tuning LLMs on Human Feedback (RLHF + DPO)

20.3K

Curating high-quality preference data is the most critical and time-consuming phase in preference-based model training. The process involves generating multiple candidate responses for each prompt, creating pairs of responses where one is clearly preferred, and manually labeling each pair. For meaningful results, this requires substantial effort—generating hundreds of candidate responses and reviewing thousands of pairs. The quality of preference data directly determines model performance, as it represents the 'programming' of the computer: telling the model what constitutes good versus bad outputs. Without careful curation, preference-tuned models may learn incorrect or suboptimal behaviors.

Click to watch →

Daniel van Strien on the Hugging Face hub and synthetic creation of a DPO dataset for Haiku

Daniel van Strien on the Hugging Face hub and synthetic creation of a DPO dataset for Haiku

305

Creating preference datasets from annotated prompts involves filtering for high-quality examples (e.g., average rating of 4 or higher with sufficient responses), then generating response pairs. The process includes: (1) filtering the prompt dataset, (2) using an LLM to generate responses for each prompt, (3) creating chosen/rejected pairs based on the generated responses. This creates the structured data needed for DPO fine-tuning, enabling models to learn from the preference patterns in the dataset.

Click to watch →

Smol course for a slow day

Smol course for a slow day

140

For Direct Preference Optimization (DPO) training, datasets typically contain structured pairs of responses—one preferred and one rejected—for each prompt. The structure includes columns for the prompt, chosen response (preferred), and rejected response. Variations may include system prompt columns or input columns containing reference material. The chosen and rejected values can be represented as strings for single-turn conversations or as conversation lists for multi-turn scenarios. This preference data structure allows the model to learn which response type is better aligned with human values by comparing the two options during training. The quality of this preference data is crucial for successful DPO implementation, requiring clear annotation guidelines to ensure consistent labeling of preferred versus non-preferred responses across diverse scenarios.

Click to watch →

Deep Dive

Evaluation methodologies for aligned LLMs, such as using LLM-as-a-judge frameworks (e.g., MT-Bench) and automated alignment benchmarks.

LLM-as-a-judge: evaluating LLMs with LLMs

LLM-as-a-judge: evaluating LLMs with LLMs

7.9K

LLM-as-a-judge is an evaluation method where large language models assess each other's outputs by classifying or scoring text against specific criteria (such as politeness, conciseness, or factual accuracy), serving as a scalable alternative to human evaluation for monitoring and improving LLM performance in production environments.

Click to watch →

LLM Evaluation Explained | Pavani R V, Microsoft

LLM Evaluation Explained | Pavani R V, Microsoft

856

Traditional metrics like BLEU, ROUGE, and F1 only measure surface-level features such as word overlap and binary correctness, failing to capture deeper qualities like factual accuracy, helpfulness, safety, and domain-specific knowledge in large language models; modern evaluation approaches address this gap through LLM-as-a-Judge methods where one LLM evaluates another using pairwise comparisons or rubric-based scoring, and through LLM-generated benchmarks that create adaptive, domain-specific tests for real-world accuracy assessment.

Click to watch →

LLM모델 성능 평가는 어떻게 진행 할까? Judging LLM as a Judge with MT Bench and Chatbot Arena 논문 리뷰!

LLM모델 성능 평가는 어떻게 진행 할까? Judging LLM as a Judge with MT Bench and Chatbot Arena 논문 리뷰!

1.4K

This paper introduces the LLM-as-a-Judge framework for evaluating large language models by using one LLM to assess another's responses, addressing the limitation that traditional benchmarks (like MMLU, HELM) only measure factual accuracy but fail to capture human-like conversational quality. The framework uses two main components: MT Bench (a multi-turn question dataset with 80 high-quality questions across eight categories) and Chatbot Arena (a crowdsourcing platform where users compare two unknown models simultaneously). Three evaluation types are proposed: pairwise comparison (selecting better answers), single-answer grading (assigning scores), and reference-based grading. Experiments show GPT-4 achieves 81% agreement with human raters, demonstrating that LLMs can reliably approximate human judgment. However, the framework has limitations including position bias (preference for first-answer position), verbosity bias (favoring longer answers), and self-serving bias (higher scores for self-generated answers). The authors propose solutions like averaging scores across positions, using multiple judges, and adding reference prompts for math problems. This approach enables scalable, rapid iteration for LLM evaluation beyond traditional automated benchmarks.

Click to watch →

LLM-as-a-Judge Evals: Comparing Kimi, Qwen, and GLM

LLM-as-a-Judge Evals: Comparing Kimi, Qwen, and GLM

692

LLM-as-a-Judge is an automated evaluation framework where one language model (the judge) assesses the outputs of another language model using detailed prompts and criteria, enabling scalable, cost-effective, and rapid comparison of models for tasks like harmful content detection, summarization quality, and prompt optimization, though it requires careful mitigation of biases such as length bias, position bias, and family bias.

Click to watch →

Evaluating LLM-based chatbots: A framework for reliable AI assistants

Evaluating LLM-based chatbots: A framework for reliable AI assistants

990

LLM evaluation employs four metric categories: pattern/syntax validation, statistical metrics (F1, BLEU, ROUGE), machine learning-based similarity measures, and LLM-as-a-judge for custom criteria. LLM-as-a-judge uses another language model to evaluate responses, supporting three variants: with reference comparison, no-reference fixed criteria, and per-question criteria. Best practices include binary scoring, detailed examples, low temperature settings, and JSON output structures. Critically, judge models must be validated against human-labeled data to ensure alignment. Multiple evaluation runs (4-5 times) are recommended due to LLM output variability, and results should be averaged rather than relying on single assessments. The goal is measuring what matters specifically for each business use case beyond generic metrics.

Click to watch →

DPO Intro

RLHF Recap

Core Loss

Reward Probabilities

Softmax Details

KL Regularization

Model Stability

Loss Function

Loss Intuition

DPO Summary

DPO Intro

0:00

Playing Section

1
Introduces Direct Preference Optimization, a method for fine-tuning LLMs without reinforcement learning.
2
Recaps the previous RLHF method and the need for a more efficient direct approach.
3
Outlines the goal to show how DPO bypasses the separate reward model training.

29.4K views86221:14@SerranoAcademyOriginal Release: 2024-06-21

Direct Preference Optimization (DPO) is a method for fine-tuning Large Language Models using human feedback without requiring reinforcement learning. Unlike Reinforcement Learning with Human Feedback (RLHF), which uses two separate neural networks (a policy network and a reward model), DPO trains only the Transformer model directly by embedding the reward function within its loss function. The Bradley-Terry model converts human reward scores into probabilities using the sigmoid function, while KL Divergence prevents the model from changing too drastically from its original behavior. This approach is more efficient than RLHF as it eliminates the need to train a separate reward model.

[00:00:00]hello I'm Louis Sano and this is Sano Academy and this video is on Direct preference optimization or DPO this is part of a series of videos in reinforcement learning so first we started with proximal policy optimization which is a way to use reinforcement learning in order to fine-tune large language models that's reinforcement learning with human feedback we also have a general video on deep re enforcement learning if you need the basics and then it was requested by the viewers to do direct preference optimization so I'm happy to bring this video to you it turns out that direct preference optimization does not use reinforcement learning so it's a direct way to find tuna large language model without the need of Po or RL it's actually a much more efficient way so it's actually able to find tuna large language model using human feedback without going through reinforcement learning it comes from this paper called direct preference optimization your model is secretly a reward model and so in this video I'm going to give you a little Refresher on what po was doing and then I'm going to show you how to bypass this using DPO are you ready let's begin so first let me give you a small recap of reinforcement learning with human feedback or rhf the idea is that we have a Transformer that we have already trained so it's already working well and we're going to use some human feedback to improve it so in order to get the human feedback we're going to feed it some prompts for example Once Upon A and then the Transformer's task is to find the next word so the next word could be time or could be banana let's say these are the two responses that the Transformer gives and it's up to a human to decide which one is better so a human evaluator will say well time is better because once upon a time is better than once upon a banana and actually the human evaluator will not just say which one's better but it'll actually give some scores so for example let's say for the sake of argument that it's going to give a score five to the good answer and a score of two to the bad answer and for the sake of this video we're going to call Scores rewards so whenever I say reward just imagine that that's the score that the human evaluator gave to a response now I've used a pretty simple prompt here but for real human evaluators they would use longer prompts longer questions so there could be two answers that are pretty good but one is simply better or more relevant to the question but for the sake of this video we're going going to use some very very simple prompts like this so how does RF work well it has two neural networks the first one is the policy neural network and the second one is the value neural network now from reinforcement learning it's common to have a policy neural network and a value neural network this time they're going to have some very specific tasks so the policy neural network is actually the Transformer model so this is the one that talks this is the model that when you input a prompt it outputs the next word and the value neuron network is called the reward model and this one learns to mimic what the human evaluator does so given a lot of responses by a human the reward model learns how a human responds and then it mimics those responses now how do we train these two networks well they help train each other so it's a cyclical process where one helps train the other one and the other one helps trains the one and here is an objective function that we saw in the PO video so if you need a refresher I invite you to check out the PO and the LF videos notice that in this formula we have X which is the prompt and then we have y w which stands for winning and that's the winning response and then y l which stands for losing which is the losing response notice that this R also appears and that refers to the reward model however there's a problem with our lhf which is that we have to train two neural networks this is a long process and it may not be very efficient all the time so we want to do directly so we're going to do away with our lhf and we're going to bring direct preference optimization and what direct preference optimization does is it only worries about one neural network that Transformer model and it kind of doesn't use the reward model so well I'm kind of lying because it doesn't fully forget it it actually embeds the reward function inside the Transformer model loss function and I'll elaborate on that so here is the loss function of the Transformer model notice a few things this pi over here is the probability of the response y given the input X so it's a probability that you would output time or output banana given that the input is 1 upon R and those Pi come from the Transformer model now I mentioned that we don't really forget about the reward model what we do is that we train the Transformer model with a loss function that includes the reward and for that we need a formula that contains the reward and the policy from the Transformer model and it's this one over here so this is going to help us get to that loss function and basically in this video I'm going to focus on this formula a lot notice that this formula has the reward over here from the reward model and the policy over here from the Transformer model you can think of that pi as the probability of the next word it could easily be called P but for some reason it's called pi here but when you see Pi think of a probability that is outputed by the Transformer model so let's focus on this formula it looks complicated but it's not so bad it has two main components the first one is this one which is trying to maximize the rewards because at the end of the day we want the Transformer neural network to Output the words or the responses that have the highest reward so for example if the input is Once Upon A and we have have time and banana we actually want it to Output time so we want to maximize that reward so that's the first part of the neural network and the second part is this one over here which looks a little cryptic but what it does is that it prevents the model from changing too drastically from one iteration to the next and I'll elaborate on why this is important but in other words what we want is to kind of get rid of the reward not get rid of it but we want to turn turn it into a probability so that the loss function only has probabilities and not rewards how do we do that using something called the Bradley Terry model I'll tell you what it is in a minute and over here we have something called dkl that is a way to measure how different two models are and that's called KL Divergence what are we comparing well we're comparing that improved model with the original model and what we're doing is comparing the distributions that these models generate because what KL Divergence does is it Compares two distributions and it outputs a large number if they're very different and a small number if they're very similar so let me first get to this part let's look at the Brader model that is going to turn rewards into probabilities so that we can rewrite this formula with only pi and not R the Bradley Terry model does the following it turns rewards into probabilities so for example if the input is Once Upon R and the outputs are time and banana well the human evaluator will give them some scores like five and two but we don't want scores we want probabilities so we're going to turn this into let's say the probability of the word time is 0.95 and the probability of the word banana is 0.05 so I'm going to show you a very simple formula that turns rewards into probabilities so let's say we have a Transformer we give it the input and it outputs two things the human evaluator says this one's good this one's bad and gives them rewards of five and two so now how would you turn these two scores into probabilities I encourage you to pause the video and think of a few ways I'm sure you can come up with a few good ones and that's why the Brad lary model provides it provides one good way of turning this course into probabilities so let's think how much would P time and P banana be and we want P of time to be higher than P of banana because time has a reward of five and banana has a reward of two so why don't we make them proportional to five and two let's say they're five and two but of course we need probabilities to add to one so let's divide by their sum which is seven and now we have 5 over 7 and 2 over 7 and those are genuine probabilities because they add to one and the probability of time is bigger than the probability of banana so are we good well no we actually need something more because this fails in some cases imagine for example that the reward was not too but -2 so all of a sudden the probability of time is 5 over 5 - 2 and the probity of banana is -2 over 5 - 2 so these two need to be positive we can't really afford negative probabilities so this model does not work but there's still a lot we can rescue any ideas on how to turn these numbers into something that is always positive well here's something that's very widely used in machine learning when you want numbers to turn into positive instead of considering the number consider e to the number so now instead of five we have e to the 5 and instead of two we have e to the two and again we want these two to add to one to be probabilities so we divide by the sum of both of those two numbers which is e to 5 plus e to 2 and when we do the math this is 0.95 over here and 0.05 over here you may have seen this before as the softmax function or even simpler as the sigmoid function this is precisely the sigmoid function and at the left we're looking at sigmoid of 5 - 2 and at the right we're looking at sigmoid of 2 - 5 if you haven't seen it or if you need a refresher here is the sigmoid function is the function 1 / 1 + e- x whose graph is this to the left is very close to zero to the right is very close to one and in the middle it switches from 0o to one and with the sigmoid function we can calculate these probabilities much quicker because what's the probability of selecting time and banana well if the reward for time is five and the reward for banana is two we simply subtract them get 5 - 2 = 3 locate the three here then locate the minus 3 here and then the probability of the winning one time is sigmoid of 3 which is 0.95 and the probability of the losing one here is p of banana which is Sigma ofus 3 or 0.05 and in general if you have a prompt X and two responses y w the good one and Y L the bad one and the reward for YW is a and the reward for y l is B the way we turn this into ities is p of Y W is sigmoid of a minus B and P of y l is sigmoid of B minus a so a minus B is over here B minus a over here and this height is p of YW sigmoid of a minus B and this height here is p of y l which is sigmoid of B minus a so that's the bradle ter model of course there are other ways to turn things into probabilities but this one has worked really well so we're going to stick with this one so before I mentioned that the formula had two main components this one over here we're trying to maximize the reward and turn into a probability and this one over here where we want to measure how different two models are which is KL Divergence so now let's focus on the second part of this formula and I'm going to tell you why we don't want the model to change too much let's imagine that we are giving our friend a test and so our friend takes the test which has 10 questions and these are their numerical response responses now our friend has studied a lot and did really well on the tests so we trust our friend however we look at question seven and we can give our friends some feedback and say hey the actual answer for question seven is this one over here so you can improve a little bit and so our friend goes and studies a little more and then comes back with this test which is very similar to the first one and it does a little better in question seven so we're happy with that this is good but then our friend says wait wait wait I did something different check this out and then it shows us a new set of responses and in here question seven is perfect now the problem is that in this test our friend changed the responses very drastically take a look at how different they are and you know we trusted our friend the first time the first time our friend did really well they fixed question seven but they changed some responses so now here is a question which one of the tests would you trust more improv test number one which does better in question seven still not great but didn't change the responses very much or improve test number two which does perfectly in question seven but it really changed the answers so I would actually go for improv test one because even though the test two did much better in question 7 it changed everything so much that it may not be trustable if the responses were good already there's no reason to change them drastically so I'm going to go go for improve tests one now how do we go for improve test one well we need a metric for how similar tests are so the original test and the improved test one are very similar however the test and the improved test two are very dissimilar so we need a way to measure how similar these tests are and that measure is going to be KL Divergence KL Divergence is actually a way to compare distributions so these two are very similar and these two are very different and what kale Divergence does is it takes a pair of distributions and returns a small number if they're very similar and a large number if they're very different the formula for K Divergence is this one over here now I won't get into detail in this video but I've made a video dedicated only to KL Divergence so if you want to take a look at that video the link is in the comments so going back to our test we're going to try the great question s in a way that rewards the test that is similar to the or original and punishes the test that is different from the original while still grading question s so we're going to come up with a modified grade for question 7 the grade is the grade in question 7 minus the Divergence with the original test so we're adding the grade in question 7 because we want to make sure the answer improved but we want to subtract the Divergence with the original test to punish tests where the answers changed a lot and reward tests where the answers stay similar and we're going to use the same rationale to find a loss function for a Transformer model for a Transformer model we're going to say that the modified grade is going to be the performance in the new data minus the Divergence with the previous model so the performance in the new data is the rewards from the data that we gave it from the new prompts and we want to measure this performance because we want to make sure the model improved on the data we gave it that's on the data that the human evaluators rated but we also want to punish the Divergence with the previous model because we want to make sure that the model does similarly on the other data that it doesn't change answers dramatically from others because we trust the model we trust the original model it's been trained already so we need to fine-tune it but we don't need to change it drastically and that's where this formula comes in because this part is the part that maximizes the rewards and this is the part that prevents the model from changing too drastically and just to be super clear the Divergence here is calculated between the Transformer model which is the one we're training and the Transformer model that we started with those are the two models we use to calculate the KL Divergence and what is this beta over here well that's a hyperparameter we can tune that hyperparameter the way we normally tune hyperparameters in machine learning and this hyper parameter tells us if we want to punish the change a lot or if we want to punish it very little in other words if we really don't want the model to change or if we're going to allow it to change a little bit in order to improve the answers so as you can see the formula looked ugly at the beginning but now it's broken into two pieces that Mak sense so now we're almost getting to the L function because we have our function on the left which maximizes rewards and punishes when the model changes too much and then we also have the Bradley Terry model which turns a reward into a probability now what they did in the paper is they put this together did a lot of mathematical manipulation and out came the loss function now notice one thing that this loss function does not have the letter r on it and that's what we wanted because we didn't really want the reward to be there directly because we don't want to be training our reward neural network now the r is there implicitly but thanks to the bradl ter model it has been turned into a probability in the Brad lary model we have it called p and in the other formula we have it called Pi but we're talking about the same probability now we can look at this loss function and say Well it it just came out of those other formulas it's a mathematical manipulation that came out of those and unfortunately that seems to be the big thing about this formula that it came out of the other three however we can still look at some things in this L function that makes sense for example check this out here we have the probability of the winning response given X is the input that means we are maximizing the probability of a good response now we can look over here and here is the probability the model gives to the bad response y l given the input X and want to minute minimize this reward why do we minimize because we have a minus sign here so we're subtracting it now notice that we also have the probabilities given by the reference model that is the original model so here is where we don't want the model to change too much now notice that we have this expected value that's because we're averaging over all the responses we have a lot of responses that we get evaluated and the loss function is the average of all these now why do we have a logarithm well I have a special thing I do when I think of logarithms when I see sums of logarithms I don't want to see them as a sum of logarithms I like to see them as the logarithm of a product and the product is normally of some numbers that are between 0 and one these are between 0 and one because you can see the sigmoid right there so we could think of them as some probability of some event and the product is the probability of all the events independently this is a trick I like to use when I see lost functions I turn the summation of logarithm into logarithm of product and then I imagine that that product has to be the probability of many things happening at the same time and now we want to maximize this probability which means we want to maximize this sum over here however a loss function we normally want to minimize it so why do we minimize it well because of this negative sign over here so we're minimizing the negative that means we're maximizing it and so it could be that one could study this formula more directly I tried for a while and pretty much the most I could get is that it was the result of a mathematical manipulation of the other formulas that are more easily explainable however I'm still thinking about it and if you have any ideas please put them in the comment I would love to be able to look at this formula and analyze every piece of it and actually make full sense of just this formula and with that we conclude so a small summary on the left we have the reinforcement learning with human feedback which used two models the Transformer model which is the policy model and the reward model which is the one that guesses the response from the human evaluator and we turn that into direct preference optimization which actually only trains one neural network the Transformer it fine-tunes it using the elaborate last function that we just derived so That's all folks thank you very much I recommend you if you haven't to take a look at all the other videos in the series they're about reinforcement learning and fine-tuning neural networks and I want to give a shout out to my friend Leticia she made a really good video on DPO and I used that video to learn a lot and to understand you should definitely check out her channel it's called AI coffee break with Leticia it's got a lot of great information especially on trendy topics on AI and really good thorough explanations so thank you very much as usual if you like this please hit like Please Subscribe and uh please add a comment I love to read the comments that people write and you know this video came out of comments because a lot of people were asking for DPO you can also check out my page Sano Academy with a lot more more information and you can tweet at me at Serano Academy and also I have a book called rocking machine learning that I encourage you to take a look at if you'd like to buy it the link is in the comments and there's also a code for a 40% discount so thank you very much and see you in the next video

Up Next

AI Engineer Portfolio Projects for Production RAG and Fine-Tuning

@aishwaryasrinivasan

56.9K views•2026-02-28

Secure Multiparty Computation (MPC): Foundations & Challenges

@SimonsInstitute

7.3K views•2015-05-28

Bypassing Tor Censorship: Bridges and Pluggable Transport Guide

@Coding_ForEveryone

397 views•2024-06-11

Neural Networks Explained: Math, Layers, and Learning Fundamentals

@3blue1brown

21.9M views•2017-10-05

Related Study Plans & Knowledge Roadmaps

Structured learning paths in Artificial Intelligence

View All Artificial Intelligence Topics

artificial-intelligence

Aligning LLMs: DPO, TRL & Custom Datasets

01Foundations of Python & Machine Learning

02Transformers and Large Language Models (LLMs)

03Aligning LLMs: SFT and RLHF

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

Federated Learning: PyTorch, Flower & Privacy

01Python & Machine Learning Foundations

02Deep Learning Basics with PyTorch

03Introduction to Federated Learning

+ 2 more learning modules

Explore Roadmap

artificial-intelligence

Securing LLMs: Guardrails AI & NeMo Gateway

01Introduction to LLMs, Python, and APIs

02Understanding LLM Vulnerabilities & Threats

03Defending LLMs with Guardrails AI

+ 2 more learning modules

Explore Roadmap

artificial-intelligence

Building Recommenders: Two-Tower & Vector DBs

01Foundations of Python & Machine Learning Basics

02Introduction to Recommender Systems

03Deep Learning & Representation Learning

+ 3 more learning modules

Explore Roadmap