Academic Databases
Explore Our Semantic Databases
STEM & Tech
Math, Physics, CS & Engineering...
Economics
Macro, Micro, Econometrics...
Business
Management, Marketing, Finance...
Health & Life Sciences
Biology, Medicine, Neuroscience...
Humanities & Social Sciences
Psychology, Law, Philosophy, History...
General Studies
General & interdisciplinary studies...
Explore Topics
Features

Online DPO Fine-Tuning for LLMs: Hands-On Implementation Guide

Added: 2026-09-10

Deep Dive

Foundational understanding of Supervised Fine-Tuning (SFT) and the standard autoregressive LLM training pipeline.

【完全解説】Karpathyの microGPT ― たった243行でGPTの全アルゴリズムを理解する

【完全解説】Karpathyの microGPT ― たった243行でGPTの全アルゴリズムを理解する

361

This section covers text generation and the complete LLM training pipeline: (1) Inference starts with BOS token and uses autoregressive generation where each output becomes the next input; (2) Temperature (T) controls diversity: T≈0 gives deterministic output, T≈1 gives diverse but potentially lower quality; (3) The three-stage pipeline includes pre-training (next-token prediction), SFT (supervised fine-tuning with Q&A pairs), and RLHF (alignment with human feedback); (4) MicroGPT demonstrates that all LLMs share the same core architecture, with differences being efficiency optimizations (parallelization, quantization, flash attention).

Click to watch →

What Is Supervised Fine-Tuning (SFT)? How ChatGPT Learns

What Is Supervised Fine-Tuning (SFT)? How ChatGPT Learns

278

Supervised Fine-Tuning (SFT) is a machine learning technique that transforms pre-trained AI models from having general knowledge into providing specific, useful answers by training them on targeted datasets. While pre-trained models learn vast amounts of information from internet data, they often produce messy or irrelevant responses. SFT addresses this by fine-tuning the model on carefully curated, task-specific data, enabling it to generate clear, accurate, and actionable responses. For example, a chatbot after SFT can provide step-by-step cake recipes instead of vague descriptions, medical AI can identify diseases from X-rays instead of describing images as 'random pictures,' and coding assistants can write proper Python functions instead of incomplete code. SFT serves as the foundational step that enables other training methods like RLHF (Reinforcement Learning with Human Feedback) to work effectively, though it requires quality training data and computational resources like GPUs.

Click to watch →

Finetuning Llama3.2 3B with an IITian | ML/LLM Project

Finetuning Llama3.2 3B with an IITian | ML/LLM Project

9.3K

This comprehensive tutorial covers the complete supervised fine-tuning (SFT) pipeline for large language models. SFT transforms pre-trained LLMs into instruction-following assistants by retraining on instruction-answer datasets, adapting models to specific tasks like document generation or PDF explanation. Three main techniques exist: Full Fine-Tuning (retraining all parameters, impractical due to resource requirements), LoRA (Low-Rank Adaptation, freezes weights and adds small adapter matrices for <1% trainable parameters), and QLoRA (Quantization-Aware LoRA, provides additional 33% memory reduction). Unsloth library offers 2x faster training and 60% memory reduction for Llama models. The pipeline involves: loading models with FastLanguageModel, creating PFT models with LoRA (rank 16), loading and standardizing datasets from Hugging Face, applying chat templates, configuring training arguments (gradient accumulation, learning rate, mixed precision), training with SFT Trainer, saving fine-tuned models, and inference with chat templates, tokenization, generation (temperature, max_new_tokens), and decoding. Key concepts include the attention mechanism (query-key-value projections), context window limits (2048 tokens), and the importance of using models with existing knowledge for the target task.

Click to watch →

Yann Lecun: Meta AI, Open Source, Limits of LLMs, AGI & the Future of AI | Lex Fridman Podcast #416

Yann Lecun: Meta AI, Open Source, Limits of LLMs, AGI & the Future of AI | Lex Fridman Podcast #416

1.3M

Autoregressive LLMs are trained by taking text, masking some words, and training a neural network to predict the masked words. The model looks only at previous words to predict the next word, creating a system that generates text one token at a time without understanding the broader context or meaning.

Click to watch →

How ChatGPT / LLM Is Built — What Nobody Explains for AI Engineers

How ChatGPT / LLM Is Built — What Nobody Explains for AI Engineers

541

Large Language Models (LLMs) like ChatGPT, Claude, and Gemini are built through a four-stage training pipeline. Stage 1 (Pre-training) involves collecting massive internet data from Common Crawl, GitHub, Wikipedia, and public books, then cleaning it through duplicate removal, language filtering, quality scoring, toxicity filtering, and PII removal. Stage 2 (Supervised Fine-tuning) teaches the model proper response structure using human-written question-answer pairs. Stage 3 (RLHF) incorporates human preferences to improve behavior and alignment. Stage 4 (Inference) is the real-time response generation. The technical foundation includes tokenization (breaking text into tokens), embeddings (converting tokens to semantic vectors), neural networks (layers capturing different information types), and transformers (with positional encodings and attention mechanisms). Training requires massive GPU resources (25,000 GPUs for ChatGPT 4.5) and costs approximately $100 million. Limitations include hallucinations (predicting likely tokens rather than truth) and knowledge cutoffs (training data is collected at a specific point in time).

Click to watch →

Deep Dive

Core concepts of Reinforcement Learning from Human Feedback (RLHF) and the standard offline Direct Preference Optimization (DPO) framework.

Direct Preference Optimization: Forget RLHF (PPO)

Direct Preference Optimization: Forget RLHF (PPO)

16.1K

Direct Preference Optimization (DPO) is a new training methodology developed by Stanford University that bypasses the complex two-step process of Reinforcement Learning from Human Feedback (RLHF) by directly optimizing language models using a single maximum likelihood objective based on human preference data, achieving similar or better performance than traditional RLHF methods while being substantially simpler to implement and train.

Click to watch →

CS 285: Eric Mitchell: Reinforcement Learning from Human Feedback: Algorithms & Applications

CS 285: Eric Mitchell: Reinforcement Learning from Human Feedback: Algorithms & Applications

8.5K

This section introduces Direct Preference Optimization as a simplified alternative to traditional RLHF. The core insight is that by parameterizing the reward function as the log probability ratio between the current policy and a reference SFT model, the optimal policy can be derived in closed form, eliminating the need for separate reward model training and RL optimization loops. The mathematical derivation shows how substituting this reward parameterization into the Bradley-Terry loss yields a tractable classification objective. Key advantages include: simplified implementation with only one policy trained; elimination of arbitrary constant degrees of freedom in reward functions; and guaranteed optimality for the fitted reward. The section explains why this works: the normalization constraint removes the constant shift problem while maintaining expressiveness. This approach achieves comparable or better performance than traditional RLHF while being computationally cheaper and more stable, representing a significant advancement in practical RLHF deployment.

Click to watch →

Direct Preference Optimization (DPO): Fine-Tune LLMs Without RL

Direct Preference Optimization (DPO): Fine-Tune LLMs Without RL

Direct Preference Optimization (DPO) is a method for fine-tuning Large Language Models using human feedback without requiring reinforcement learning. Unlike Reinforcement Learning with Human Feedback (RLHF), which uses two separate neural networks (a policy network and a reward model), DPO trains only the Transformer model directly by embedding the reward function within its loss function. The Bradley-Terry model converts human reward scores into probabilities using the sigmoid function, while KL Divergence prevents the model from changing too drastically from its original behavior. This approach is more efficient than RLHF as it eliminates the need to train a separate reward model.

Click to watch →

Have you heard these exciting AI news? - December 22, 2024 - AI Updates Weekly

Have you heard these exciting AI news? - December 22, 2024 - AI Updates Weekly

2.2K

RLHF (Reinforcement Learning from Human Feedback) is the traditional approach for fine-tuning LLMs, involving four steps: pre-training, generating question-answer pairs, collecting human feedback, and training a reward model to mimic human preferences before using it to train the main model. DPO (Direct Preference Optimization) simplifies this by directly using human feedback to train the main model without an intermediate reward model, removing the reinforcement learning component entirely. DPO is simpler, cheaper, more stable, and eliminates the need for the reward model.

Click to watch →

PR-453: Direct Preference Optimization

PR-453: Direct Preference Optimization

5.3K

This section introduces Direct Preference Optimization (DPO) as an alternative to Reinforcement Learning from Human Feedback (RLHF) for aligning Large Language Models with user intent. The motivation stems from challenges in training LLMs on massive datasets containing billions of tokens, which include undesirable content like misconceptions and errors. While human demonstrations through supervised fine-tuning can address quality issues, preference-based reinforcement learning has proven most successful. However, RLHF requires training both a language model policy and a reward model, then performing on-policy reinforcement learning, which is computationally expensive and complex. The section details the three-stage RLHF process: Supervised Fine-Tuning (SFT), Reward Modeling using Bradley-Terry or Plackett-Luce models with ranking loss, and Policy Optimization using PPO. Major industry implementations include OpenAI's InstructGPT/ChatGPT, Anthropic's Claude, DeepMind's Sparrow, and Meta's Llama 2, each with variations in response sampling, reward models, and training strategies.

Click to watch →

Deep Dive

Familiarity with the Hugging Face ecosystem, specifically the 'transformers', 'peft' (for LoRA), and basic 'trl' libraries.

LLM Crash Course Part 1 - Finetune Any LLM for your Custom Usecase End to End in under[1 hour]!!

LLM Crash Course Part 1 - Finetune Any LLM for your Custom Usecase End to End in under[1 hour]!!

3K

This section covers setting up the environment for LLM fine-tuning: (1) Installing required dependencies including Transformers 4.37.2, TRL, PEFT, Accelerate, datasets, and bitsandbytes, (2) Configuring Hugging Face token for accessing Hub resources, (3) Understanding the key inputs needed for fine-tuning including torch, datasets, PEFT, transformers, and TRL libraries. The video emphasizes that specific versions are needed for compatibility with Google Colab's environment.

Click to watch →

Fine-tuning open AI models using Hugging Face TRL

Fine-tuning open AI models using Hugging Face TRL

13.9K

Fine-tuning is a supervised learning technique that improves a pre-trained language model's performance on specific tasks by training it on task-specific examples; using Hugging Face TRL with Parameter Efficient Fine-Tuning (PEFT), you can create specialized adapters that add minimal weights to a base model, allowing even modest GPUs to efficiently train models like Gemma 2 for tasks such as code generation, while producing compact adapter files that can outperform larger general-purpose models on those specific tasks.

Click to watch →

LoRA explained (and a bit about precision and quantization)

LoRA explained (and a bit about precision and quantization)

118.5K

The Hugging Face library provides easy implementation of LoRA through the PEFT (Parameter Efficient Fine-Tuning) repository. Instead of manually applying rank decompositions to every layer, developers use get_peft_model function. Configurations allow specifying target modules (like QKV matrices in Transformers) and hyperparameters alpha and rank. An example shows only 0.19% of original model weights are trained, enabling training of huge models on single GPUs.

Click to watch →

Medical Chatbot with Qwen2.5 | Fine-Tuning LLM Using LoRA | Hugging Face Deployment | Gradio UI

Medical Chatbot with Qwen2.5 | Fine-Tuning LLM Using LoRA | Hugging Face Deployment | Gradio UI

541

This comprehensive project demonstrates building a medical chatbot through a complete pipeline: (1) Project Overview - fine-tuning a 0.5B parameter LLM using LoRA for parameter-efficient training; (2) Dataset Preparation - converting Gale Encyclopedia of Medical Science into structured CSV with user instructions and responses; (3) Technical Implementation - using Transformers, PEFT, TRL, and Gradio libraries with 4-bit model loading; (4) Training Process - tokenization, 3 epochs with 624 steps, TensorBoard monitoring showing loss reduction from 0.38% to 0.16%; (5) Deployment - deploying on Hugging Face Spaces for real-time user interaction. The project demonstrates how to create a healthcare chatbot that generalizes to new queries using minimal computational resources.

Click to watch →

Efficient Large Language Model training with LoRA and Hugging Face PEFT

Efficient Large Language Model training with LoRA and Hugging Face PEFT

7.6K

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique that trains only a small subset of a large language model's weights (less than 1%), dramatically reducing computational costs and training time compared to full fine-tuning; when combined with Hugging Face's PEFT library and bitsandbytes quantization (reducing memory usage by ~4x), this approach enables efficient fine-tuning of billion-scale models like Flan-T5 XXL on consumer-grade GPUs, achieving competitive performance with significantly lower resource requirements.

Click to watch →

Deep Dive

Understanding the structure of preference datasets, which typically contain prompt, chosen, and rejected response triplets.

LLM Fine-Tuning 16: Preference Alignment & Preference Training in LLMs with RLHF, RLAIF, DPO, LoRA

LLM Fine-Tuning 16: Preference Alignment & Preference Training in LLMs with RLHF, RLAIF, DPO, LoRA

1.7K

Preference training datasets contain three main columns: (1) Prompt - the input question or instruction given to the model, (2) Chosen - the response that should be selected/preferred, and (3) Rejected - the response that should be rejected. Examples include datasets from Hugging Face such as 'anthropic/hh_rlhf' and 'ultrafeedback_binarized_clean', which provide ranked responses where human annotators have labeled which response is better.

Click to watch →

From RLHF with PPO/DPO to ORPO + How to build ORPO on Trainium/Neuron SDK

From RLHF with PPO/DPO to ORPO + How to build ORPO on Trainium/Neuron SDK

917

Preference optimization datasets typically contain three key columns: (1) Prompt - the user question or input, (2) Chosen response - the correct or preferred answer that should be generated, (3) Rejected response - an alternative output that the model might otherwise produce but is considered suboptimal. For example, for the prompt 'What time is it?', the chosen response might be 'It is 9:45 AM PST,' while rejected responses could include 'Nothing you can do about hot houses' (unhelpful) or 'It's not too hot' (disagreeing with the premise). This structure enables direct comparison and training on what outputs should be favored.

Click to watch →

ML Paper Reading Club #1: Self-Rewarding Language Models | GDG Hong Kong

ML Paper Reading Club #1: Self-Rewarding Language Models | GDG Hong Kong

116

A preference dataset consists of three components for each example: (1) the instruction prompt, (2) the winning (chosen) response that should be preferred, and (3) the losing (rejected) response that should be avoided. These preference pairs are constructed by comparing scored responses and selecting the higher-quality response as the chosen one, providing structured training data for preference-based learning algorithms.

Click to watch →

Aligning LLMs with Direct Preference Optimization

Aligning LLMs with Direct Preference Optimization

33.9K

DPO datasets require prompts, chosen responses, and rejected responses. Not all datasets provide binarized preferences; some offer ranked responses. Best practice involves selecting the highest-ranked response as 'chosen' and randomly choosing another as 'rejected' to create diverse training examples with varying margins. Multi-turn conversations may have only the final response differing between chosen and rejected paths. The DPO trainer handles linked conversations during loss calculation.

Click to watch →

RLHF in 90 min

RLHF in 90 min

4.4K

Building a preference dataset involves a four-step pipeline: (1) Take a prompt from the dataset; (2) Use the SFT model to generate several different responses (A, B, C, D); (3) Show these responses to human labelers who rank them from best to worst; (4) Convert the ranking into pairwise comparisons. For example, a ranking B > D > A > C generates multiple data points including (prompt, B, D), (prompt, B, A), and (prompt, D, A). This creates a massive dataset of tuples containing prompts, winner responses, and loser responses.

Click to watch →

Intro & Concepts

Online DPO Explained

Setup & Install

Model Training

Results & Review

Intro & Concepts

0:01

Playing Section

1
Explains fine-tuning and how models are aligned with personal data.
2
Compares DPO with traditional PPO, highlighting efficiency and stability gains.

Deep Dive

Exploring alternative and advanced alignment algorithms such as Kahneman-Tversky Optimization (KTO), Identity Preference Optimization (IPO), and Odds Ratio Preference Optimization (ORPO).

LLM Alignment Methods - DPO vs IPO vs KTO vs PCL

LLM Alignment Methods - DPO vs IPO vs KTO vs PCL

1.6K

This video introduces four modern methods for aligning large language models with human preferences: Direct Preference Optimization (DPO) recasts alignment as a simple loss function using preference pairs; Identity Preference Optimization (IPO) adds regularization to prevent overfitting; Kahneman-Tversky Optimization (KTO) uses individual example labels instead of pairwise preferences; and Pairwise Cringe Loss offers an efficient alternative that outperforms competing methods. DPO is recommended as the most practical starting point due to its simplicity and effectiveness.

Click to watch →

Reinforcement Learning, RLHF, & DPO Explained

Reinforcement Learning, RLHF, & DPO Explained

15.4K

This video explains three optimization techniques for aligning AI language models with human preferences: (1) RLHF uses reinforcement learning where a reward model scores outputs and an optimizer updates the policy model through trial and error, but faces challenges with instability and expensive human labeling; (2) DPO directly optimizes the language model on preference data without explicit reward modeling, achieving equivalent or better results than RLHF while being simpler, faster, and more robust to temperature changes; (3) KTO applies prospect theory from behavioral economics to maximize utility by considering human risk aversion, using a sigmoid value function and handling noisy/inconsistent data better than other methods, with tunable hyperparameters for balancing risk aversion between desirable and undesirable examples.

Click to watch →

4 Ways to Align LLMs: RLHF, DPO, KTO, and ORPO

4 Ways to Align LLMs: RLHF, DPO, KTO, and ORPO

3.9K

This video explains four key methods for aligning large language models to enterprise needs: (1) Reinforcement Learning with Human Feedback (RLHF) uses a reward model trained on human-labeled preference data to guide model optimization; (2) Direct Preference Optimization (DPO) eliminates the separate reward model by directly optimizing the LLM using preference data; (3) Kahneman-Tversky Optimization (KTO) applies prospect theory principles using binary labels for robustness against label noise; and (4) Odds-Ratio Preference Optimization (ORPO) combines supervised fine-tuning and preference optimization into a single computationally efficient loss function. The video emphasizes that well-defined preference dimensions and avoiding bias in human/model-generated rankings are critical for successful alignment outcomes.

Click to watch →

ORPO: Monolithic Preference Optimization without Reference Model (Paper Explained)

ORPO: Monolithic Preference Optimization without Reference Model (Paper Explained)

25.7K

ORPO (Odds Ratio Preference Optimization) is a unified training framework that integrates supervised fine-tuning and preference alignment into a single joint loss function, eliminating the need for separate reference models or multi-step training processes; it addresses the limitation of standard supervised fine-tuning by using an odds ratio loss that distinguishes between winning and losing responses, thereby improving alignment performance while maintaining instruction-following capabilities.

Click to watch →

LLM Fine-Tuning: Exploring RLHF Alternatives

LLM Fine-Tuning: Exploring RLHF Alternatives

278

This video explains three key LLM fine-tuning methods: Reinforcement Learning with Human Feedback (RLHF), which uses human feedback to train a policy model for response selection; Direct Preference Optimization (DPO), which simplifies RLHF by converting pairwise preferences into probabilities using Bradley-Terry methodology but struggles with deterministic tasks; and newer alternatives like Identity Preference Optimization (IPO) and Kahneman-Tversky Optimization (KTO), which address DPO's limitations. The presenters recommend starting with prompt engineering and few-shot prompting before progressing to fine-tuning, and suggest breaking down complex tasks into modular components to improve performance without extensive retraining.

Click to watch →

Deep Dive

Implementing iterative alignment loops and self-rewarding paradigms, where the model generates its own training data and critiques it.

Adv. LLM Agents MOOC | UC Berkeley CS294-280 Sp25 | Learning to Reason with LLMs by Jason Weston

Adv. LLM Agents MOOC | UC Berkeley CS294-280 Sp25 | Learning to Reason with LLMs by Jason Weston

14.1K

This section explains self-rewarding language models, where LLMs improve themselves by assigning rewards to their own outputs. The motivation is that as models become superhuman, humans struggle to provide good judgments for specialized tasks. The approach combines: (1) language models can continue improving with good judgments, and (2) language models can provide good judgments on model generation. The iterative training loop involves: generating new instructions via self-instruct, generating candidate responses, using LLM as a judge to score responses, creating DPO preference pairs, and repeating. Demonstrated on Llama 70B, this approach improved win rates from 10% to 20% on AlpacaEval2 across three iterations, approaching GPT-4 performance.

Click to watch →

Researchers STUNNED As A.I Improves ITSELF Towards Superintelligence (BEATS o1)

Researchers STUNNED As A.I Improves ITSELF Towards Superintelligence (BEATS o1)

122K

The model generates its own training data through a feedback loop: it generates solutions, evaluates them, and retrains on the best solutions, then repeats. For problems it cannot solve, the model uses multiple reasoning attempts with additional rollouts or new random seeds until it finds a solution. This ensures the model learns from its failures. After the initial round, the model generates its own training data, which is a major shift away from traditional training methods that rely heavily on distillation from larger teacher models.

Click to watch →

Meta's Shocking New Research | Self-Rewarding Language Models

Meta's Shocking New Research | Self-Rewarding Language Models

70.3K

Experimental results demonstrate consistent improvement through iterative self-rewarding training. Baseline comparisons show M1 ≈ Baseline, M2 wins 50% vs Baseline (15% win rate), and M3 wins 62% vs Baseline (<10% win rate). Cross-iteration comparisons show M3 beats M1 70% of the time. On Alpaca Eval 2.0, M3 achieves 2.44% win rate over GPT-4 Turbo. Reward modeling ability also improves progressively across iterations, validating the effectiveness of self-rewarding approaches for creating models that can continually improve beyond human-preference-limited training.

Click to watch →

Reinforced Self-Training (ReST) for Language Modeling (Paper Explained)

Reinforced Self-Training (ReST) for Language Modeling (Paper Explained)

34.4K

A naive self-training approach fails because training on self-generated data cannot improve a model—it merely stabilizes at current performance ('eating your own poop'). For self-training to succeed, a reward model is essential to distinguish good from bad generated data. This reward model acts as the critical filter that enables progressive improvement. The paper reveals that the core missing ingredient in many self-training approaches is the reward model, which must be sufficiently accurate to guide the iterative improvement process.

Click to watch →

The Most Powerful ChatGPT Prompts Right Now

The Most Powerful ChatGPT Prompts Right Now

174.2K

The reflection self-critique loop involves drafting an answer, then identifying logical errors, gaps, and fallacies in that draft, and producing an improved final answer that fixes all identified issues. This technique enhances accuracy by forcing the model to examine and correct its own reasoning.

Click to watch →

Deep Dive

Evaluating aligned models using standardized LLM benchmarks (e.g., MT-Bench, AlpacaEval) and LLM-as-a-judge evaluation frameworks.

Building Large Language Models: Stanford CS229 Lecture

Building Large Language Models: Stanford CS229 Lecture

Evaluating aligned language models differs from pre-trained models because aligned models optimize for behaviors rather than maximum likelihood, making perplexity unreliable. Chatbot Arena uses real users to rate competing models, while Alpaca Eval uses language models to compare outputs, achieving 98% correlation with human judgments at lower cost. Challenges include preference annotation costs ($300/1,000 examples), inter-rater reliability (only 66% agreement), and distribution shifts affecting results.

Click to watch →

LLM모델 성능 평가는 어떻게 진행 할까? Judging LLM as a Judge with MT Bench and Chatbot Arena 논문 리뷰!

LLM모델 성능 평가는 어떻게 진행 할까? Judging LLM as a Judge with MT Bench and Chatbot Arena 논문 리뷰!

1.4K

This paper introduces the LLM-as-a-Judge framework for evaluating large language models by using one LLM to assess another's responses, addressing the limitation that traditional benchmarks (like MMLU, HELM) only measure factual accuracy but fail to capture human-like conversational quality. The framework uses two main components: MT Bench (a multi-turn question dataset with 80 high-quality questions across eight categories) and Chatbot Arena (a crowdsourcing platform where users compare two unknown models simultaneously). Three evaluation types are proposed: pairwise comparison (selecting better answers), single-answer grading (assigning scores), and reference-based grading. Experiments show GPT-4 achieves 81% agreement with human raters, demonstrating that LLMs can reliably approximate human judgment. However, the framework has limitations including position bias (preference for first-answer position), verbosity bias (favoring longer answers), and self-serving bias (higher scores for self-generated answers). The authors propose solutions like averaging scores across positions, using multiple judges, and adding reference prompts for math problems. This approach enables scalable, rapid iteration for LLM evaluation beyond traditional automated benchmarks.

Click to watch →

[Lab Seminar] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

[Lab Seminar] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

165

This paper introduces LLM-as-a-Judge, an automated evaluation framework that uses large language models (such as GPT-4) to evaluate model responses and judge alignment with human preferences, addressing the limitations of existing benchmarks that cannot distinguish between aligned and unaligned models in multi-turn conversations; the framework achieves over 80% agreement with human evaluators, demonstrating that powerful LLMs can serve as reliable automated judges for evaluating open-ended questions and human preference tasks.

Click to watch →

LLM-as-a-judge: evaluating LLMs with LLMs

LLM-as-a-judge: evaluating LLMs with LLMs

7.9K

LLM-as-a-judge is an evaluation method where large language models assess each other's outputs by classifying or scoring text against specific criteria (such as politeness, conciseness, or factual accuracy), serving as a scalable alternative to human evaluation for monitoring and improving LLM performance in production environments.

Click to watch →

Stanford CS229 I Machine Learning I Building Large Language Models LLMs

Stanford CS229 I Machine Learning I Building Large Language Models LLMs

113

Evaluating aligned models differs fundamentally from pre-trained models: perplexity is unreliable (models are trained as policies, not maximum likelihood models), and validation loss isn't comparable across approaches. Solutions include: (1) Human preference evaluation via platforms like Chatbot Arena collecting thousands of user judgments; (2) Model-based evaluation using other LLMs as judges (achieving ~98% correlation with human judgments at lower cost). LM judges can introduce biases (preferring longer responses), requiring causal analysis to control confounding factors. These evaluation approaches enable meaningful comparison across models despite architectural and training differences.

Click to watch →

Deep Dive

Scaling online DPO training to multi-GPU setups using distributed training libraries like DeepSpeed, FSDP, or Ray.

Webinar: Getting Started with Distributed Training at Scale

Webinar: Getting Started with Distributed Training at Scale

507

Distributed training enables scaling machine learning workloads beyond single-GPU limitations by distributing data across multiple GPUs or machines, with techniques ranging from simple Data Parallel (DDP) where each GPU holds a full copy of the model and shards gradients through all-reduce communication, to advanced ZeRO-1/2/3 and FSDP methods that progressively shard optimizer states, gradients, and parameters to reduce memory overhead; Ray Train simplifies this process by abstracting away the complexity of PyTorch's distributed training stack through APIs like prepare_model() and prepare_data_loader(), allowing developers to scale training from single-GPU to multi-node clusters with minimal code changes while automatically handling distributed initialization, data partitioning, and checkpoint management.

Click to watch →

Multi GPU Fine tuning with DDP and FSDP

Multi GPU Fine tuning with DDP and FSDP

17.7K

This comprehensive section provides hands-on guidance for implementing multi-GPU training systems. It covers practical deployment including setting up SSH access to rented GPU instances (like RunPod), cloning training repositories, installing dependencies (PyTorch, Transformers, Accelerate, DeepSpeed, FlashAttention), configuring GitHub credentials, and using accelerate config to set up distributed training parameters including sharding strategy, wrapping policies, and data types. The section details code modifications for Distributed Data Parallel (replacing device_map, running with accelerate launch, setting use_reentrant=False, applying gradient checkpointing) and Fully Sharded Data Parallel (not passing device_map, setting use_reentrant=True, applying custom wrapping policies, gathering state dictionary before saving). Experimental results demonstrate significant speedups: DDP provides ~4x speedup over model parallel for Tiny Llama, while FSDP provides ~1.5x speedup over model parallel for Code Llama 34B, though FSDP has higher communication overhead. The section emphasizes that more GPUs generally introduce more problems, and ideally one GPU should be used if possible.

Click to watch →

Build LLMs From Scratch with Sebastian Raschka #52

Build LLMs From Scratch with Sebastian Raschka #52

6.2K

PyTorch Lightning is a wrapper library around PyTorch that simplifies distributed training by handling boilerplate code for multi-GPU setups, mixed precision training, checkpointing, and logging. Similar to how Keras simplifies TensorFlow, PyTorch Lightning makes PyTorch easier to use for scaling models beyond single-GPU setups. Users can switch between single and multiple GPUs with minimal code changes by specifying a strategy (like FSDP or DeepSpeed) and the number of GPUs. The library has approximately 30k stars on GitHub, indicating widespread adoption.

Click to watch →

PyTorch Lightning #10 - Multi GPU Training

PyTorch Lightning #10 - Multi GPU Training

10.5K

PyTorch Lightning simplifies multi-GPU training through built-in strategies like Data Parallel (DDP) for basic distributed training and DeepSpeed for handling large models that exceed single-GPU VRAM limits; DDP copies the model to all GPUs and processes data in parallel, while DeepSpeed's Zero algorithm stages (1-3) progressively optimize memory usage by offloading to CPU or splitting model memory across GPUs, enabling efficient scaling of training workloads.

Click to watch →

Lec 13 | Efficient LLMs: Part 03

Lec 13 | Efficient LLMs: Part 03

434

DeepSpeed provides FSDP (Fully Sharded Data Parallel) for sharding models across GPUs. PyTorch's native FSDP support allows sharding models layer by layer. Distributed samplers split datasets across GPUs using world_size and rank. Gradient accumulation enables processing multiple micro-batches before updating weights. Gradient checkpointing reduces activation memory by recomputing activations during backward pass instead of storing them.

Click to watch →

Intro & Concepts

Online DPO Explained

Setup & Install

Model Training

Results & Review

Intro & Concepts

0:01

Playing Section

1
Explains fine-tuning and how models are aligned with personal data.
2
Compares DPO with traditional PPO, highlighting efficiency and stability gains.

768 views2611:11@fahdmirzaOriginal Release: 2024-09-02

Online DPO (Direct Preference Optimization) is an alignment method that fine-tunes large language models by generating preference data on-the-fly during training, using a reward model to select preferred completions for each prompt, which eliminates the need for pre-collected preference datasets and enables continuous model improvement while achieving better results than traditional DPO.

[00:00:01]hello everyone and welcome to the channel I'm very excited to report that now hugging face celebrated TRL Library supports yet another fine-tuning method called as online DPO or online direct preference optimization before I show you the Hands-On demo as how to get this DRL installed and then how to do the online dpu on your own data set let me first set the stage by explaining what exactly is meant by fine-tuning what is this online DPO and why this is such a important Milestone first and foremost TRL is a library from hugging phase which is a full stack library that provides a set of tools to train transformer language models with reinforcement learning from supervised fine tuning to reward modeling to various other things now let's try to simplify these Concepts what exactly is fine-tuning in simple words what happens is that all of these large language models no matter what the model is they are trained on a huge generic set of data but they that data and that model doesn't really reflect your own data your own requirements if you want to align the model as per your own preferences for example you just want the model to always align with your values your goals your preferences then you can create a data set and then F tune or retrain that model as per your own data set so that for example you don't like Winters so you can train the model on that item so that whenever someone asks the model about winter model would say that model doesn't like winter because you have given that preference to the model and you have train the model on that data set that is a very high level example of course but you get the point and that is what called as direct preference optimization where you give it a preference now for the fine tuning there are lot of methods for example if you look at this diagram and by the way this and this diagram I have taken from Maxim labone who a really um good researcher out there and I have interviewed him for my channel if you're interested just search for his name on the channel Okay so now what exactly here is happening is that this is a diagram for the PO algorithm which is the proximal policy optimization now don't get too much worried about the naming here so all what it is happening here is that we are fine-tuning the models from Human preferences that is what is generically at high level called as uh reinforcement learning with human feedback where they have presented this framework where a reward model is trained to approximate human feedback and then this reward model is used to optimize the fine-tuned models policy using the proximal policy algorithm now the core Concept in this diagram is revolving around making smaller incremental updates to the policy as larger updates can lead to instability the problem where is that there is lot of loss time convergence it is difficult to reproduce and it is quite expensive to be honest that is where a new technique emerged a few months back called as direct preference optimization or DPO DPO simplifies this by treating the task as a classification problem it only uses two models train model or a copy of it called the reference model during the training the goal is to make sure that the train model outputs higher probabilities for preferred answers than the reference model and we also wanted to Output lower probabilities for rejected answer so what it what happens is that we give it a prompt we give it a preferred answer and a rejected answer and then we find you the model so that model always know what answer is the acceptable one as per her own alignments and preferences so that is what uh direct preference optimization is now recently and very very recently this new technique has emerged called as online DPO or online direct preference optimization it's a new alignment method from Deep Mind to boost the performance of llms with online DPO data is generated on the Fly by the train model instead of pre-collected for each prompt two completions are generated with a reward model selecting the preferred one what this approach does is it eliminates the need for a pre-collected preference data set it is ated online and that is really amazing it also enables continuous model Improvement and it yields better results from a traditional DPO so that is what has been now merged in this hugging faces TRL library and that is what we are going to do in this video we will install TRL library from The Source from the recent branch and then we will see how to fine tune a model on online uh DPO data set before I do that let me give a huge shout out to M compute who are sponsoring the VM and GPU for this video If you're looking to rent a GPU on affordable prices I will drop the link to their website in video's description plus I'm also going to give you a coupon code of 50% discount on range of gpus so this is my one2 terminal with 22.4 and this is my GPU card nvd RTX a6000 okay so let me clear the screen first up let me create a virtual environment with Konda let's wait for it to get activated the virtual environment is created let's install all the prerequisites here I'm installing T torch Transformers and lot of other stuff so let's wait for it to finish and don't worry about the code I'm also going to provide you the link to the code which I'm using from start to end all the prerequisites are done next up after clearing the screen let's get clone the repo of TRL and the reason why I am doing it is that this feature is very very new so we have to check out the recent Branch where they have committed this if you're using it like after maybe couple of weeks then you might not have to do it you can install it from The Source but at uh it I don't think so it has been merged into the main branch so that is why I have switched to that uh commit HSA so let me install it from that Branch let's wait for it shouldn't take too long and these are the joys of working at the bleeding edge anyway so it is installed now let's Now launch our Jupiter notebook because that is where I'm going to install it and then see how it works so let's wait for it to launch it in the browser and then I will show you how you can use this Dr library to do online DPO on your own data set so what jupyter notebook is launched let me paste the code here now what this code is doing it is simply importing the libraries which we have installed here you can see and then it is just defining some of the dummy samples grabbing our tokenizer grabbing our model I'm just going to go with this small LM with 135 million parameter just to make things faster and easier and then we are defining so first up this is the model which we are use looking to optimize or fine tune and then this is the reference model same model and this is going to calculate the K Divergence again K Divergence measures how different two probability distributions are and U this mainly helps a model understand uncertainity and similarity okay so once that's done we are defining this reward model to score completions with of course uh you will definitely need a reward model for online dpu as I explained earlier and this is whereever training and eval data set is I'm just giving it this prompt and simple but of course you can just just replace it with your own data set here and then we are just calling our online DPO trainer with the model reference reort R tokenizer and all the usual stuff and then we are running the training so once I start it you will see it is going to prepare our data set and then from there it is going to map it and then get our models there you go so it has started the training it's a very small data set so it should be fairly quick I believe and this is how you do the online DPO but the main thing here is that to understand the reward model and this stuff and then getting your data set and this is how easy it becomes once you understand that uh how you should prepare your data set how should you just put a reward model and of course you can select different models for this one maybe you could have Lama 3.1 70 billion1 as a reward model so that your performance of the F your model would be more Superior and then there you go the fine tuning is done it is giving you the statistics around it that what was a training loss which has come down how much was a train runtime aox and all that stuff now if you want to see your new model which is fine tuned just go to your um terminal maybe I'll just open a new one and then I just deactivate my original base model base gond and then go to TRL if you do LS here you will see that there should be directory called as online DPO model if you do LS there should be a check point yep and this is your model file with all the tokenizer config do GSN and your models save ters here and then there's also some other Json files with vocabulary and stuff so this is how you create a new findu model from your own data set by using this online deep which seems quite promising new fine tuning technique so that's it I hope that you enjoyed it let me know what do you think if you like the content please consider subscribing to the channel thanks for watching

Up Next

Introduction to Secure Multiparty Computation with Yehuda Lindell

@fhe_org

7.7K views•2021-02-04

Supervised vs Unsupervised Learning: ML Foundations

@digiLab_ai

108 views•2023-06-22

HTTP Requests Explained: GET, POST, PUT, DELETE

@codecademy

103.1K views•2021-10-07

Enigma Machine Mechanics: WWII Encryption Explained

@JaredOwen

13.2M views•2021-12-11

Related Study Plans & Knowledge Roadmaps

Structured learning paths in Computer Science

View All Computer Science Topics

computer-science

Container Runtimes: Namespaces, Cgroups & Go

01Linux Systems & Go Programming Foundations

02The Core of Isolation: Chroot and Filesystems

03Linux Namespaces: Process Isolation

+ 3 more learning modules

Explore Roadmap

computer-science

Building a CHIP-8 Emulator: Rust & CPU Basics

01Computer Architecture Foundations (Bits & Bytes)

02Rust Programming for Systems Development

03CHIP-8 Virtual Hardware Specifications

+ 2 more learning modules

Explore Roadmap

computer-science

Building BitTorrent: Go, P2P & Concurrency

01Go Fundamentals & Concurrency

02Networking Foundations & P2P Architecture

03Bencode Parsing & Torrent Metadata

+ 3 more learning modules

Explore Roadmap

computer-science

Actor Models (C++ Concurrency & Lock-Free)

01C++ Multi-Threading Foundations

02Thread Pools and Task Scheduling

03Atomics and Lock-Free Queue Design

+ 3 more learning modules

Explore Roadmap