Aligning LLMs: DPO, TRL & Custom Datasets

Learning Goal: Align a pre-trained large language model (LLM) with human preferences using Direct Preference Optimization (DPO) and the TRL library on a custom feedback dataset.

Prerequisites

• Basic intermediate programming experience in Python (variables, control loops, standard data structures).
• Introductory knowledge of machine learning concepts (features, labels, training/loss, gradient descent).
• Access to a GPU-enabled notebook environment (such as Google Colab, Kaggle, or a local CUDA setup).

Estimated Study Time: 24 Hours

---

Course Map

---

Module 1: Foundations of Python & Machine Learning

Understand basic Python syntax, data types, and core machine learning paradigms. This module bridges standard programming skills with foundational statistical concepts such as supervised learning and optimization.

Recommended Videos

Why this video: This crash course is essential for establishing strong programming foundations. It quickly catches you up to speed on variable typing, data structures (especially lists and dictionaries), control flow (if/else), loops, and basic functions, all of which are vital for preparing data pipeline scripts in Python.

---

[3/11] Supervised vs unsupervised learning - Foundations in Machine Learning | digiLab Academy

Why this video: This quick video provides a clear conceptual bridge into ML. It highlights the difference between learning from labeled input-output pairs (supervised learning, which powers language models and alignment steps) and discovering latent structures without labels (unsupervised learning).

---

Why this video: Deep learning models generate predictions using probability distributions. This video establishes the necessary mathematical language for machine learning optimization, showing why linear algebra, calculus, and probability are the bedrocks of neural net modeling.

Knowledge Checkpoint

• Write a basic Python script that parses a list of raw text lines and structures them into a list of dictionaries.
• Describe the difference between unsupervised pre-training (next-token prediction) and supervised fine-tuning.
• Define what a probability distribution is and explain how it maps to model predictions.

---

Module 2: Transformers and Large Language Models (LLMs)

Grasp how modern neural networks process sequence data using the Transformer architecture, and learn how pre-trained LLMs generate text token by token.

Recommended Videos

Why this video: An outstanding conceptual deep-dive into how the Transformer architecture revolutionized natural language processing. It breaks down the shift from older recurrent models (RNNs/LSTMs) to self-attention mechanisms, explaining how models process entire text sequences in parallel.

---

Why this video: This hands-on tutorial gets you comfortable with Hugging Face's foundational APIs. It details how pipeline abstractions, model configurations, and tokenizers work in Python, preparing you for the lower-level API configurations used in fine-tuning.

---

Why this video: Focusing on the practical code segment of tokenization, this video shows how tokenizer.encode_plus actually takes text strings and outputs numerical representations (input IDs, attention masks, and special tokens like [CLS] and [SEP]) that are directly consumed by transformer models.

Knowledge Checkpoint

• Explain how self-attention allows a model to weigh the importance of different words in a sentence.
• Implement a basic script that initializes a Hugging Face tokenizer, converts a raw text string into token IDs, and decodes it back.
• Define the purpose of an attention mask in sequence processing.

---

Module 3: Aligning LLMs: SFT and RLHF

Before diving into DPO, you must understand the traditional alignment pipeline: Supervised Fine-Tuning (SFT) followed by Reinforcement Learning from Human Feedback (RLHF). This module teaches the historical and conceptual reasons why alignment is necessary to make LLMs helpful, honest, and harmless.

Recommended Videos

Why this video: An incredibly clear, step-by-step layout of RLHF. It breaks down how candidate responses are generated, how human preferences are collected to train a distinct "Reward Model," and how Proximal Policy Optimization (PPO) fine-tunes the base model's parameters using that reward signal.

---

Why this video: Alignment is not just a coding problem; it is a safety imperative. This video discusses the philosophical and practical challenges of ensuring high-capacity AI systems adhere to human intent, providing a conceptual backdrop for why RLHF and DPO are studied.

---

Why this video: This video helps you classify when to fine-tune your model versus when to use Retrieval-Augmented Generation (RAG). Alignment changes the style, tone, behavior, and structural rules of your model, whereas RAG dynamically pulls in external, factual knowledge.

Knowledge Checkpoint

• Explain the three standard stages of building a chat assistant: Pre-training $\rightarrow$ Supervised Fine-Tuning (SFT) $\rightarrow$ Alignment (RLHF/DPO).
• Describe how a "Reward Model" is trained on binary preferences (chosen vs. rejected responses) to score model completions.
• Distinguish when to apply alignment fine-tuning versus when to use an in-context RAG pipeline.

---

Module 4: Direct Preference Optimization (DPO)

Learn how Direct Preference Optimization (DPO) bypasses the need for training a separate reward model or running complex reinforcement learning loops (PPO), simplifying alignment down to a binary classification task.

Recommended Videos

Why this video: This is the definitive mathematical breakdown of the DPO paper. Umar Jamil masterfully explains the Bradley-Terry preference model, details how log probabilities of policy and reference models are computed, and steps through the analytical proof showing how the loss function substitutes the reward model out of the system.

---

Why this video: An academic overview highlighting the architectural differences between DPO and RLHF. It reviews why DPO requires maintaining fewer active models in memory (only the policy and reference model) and touches on stable hyperparameter tuning.

---

Why this video: A brief, intuitive visualization of how DPO takes a prompt, raises the generation probability of the preferred ("chosen") answer, and suppresses the generation probability of the non-preferred ("rejected") answer simultaneously.

Knowledge Checkpoint

• Explain how DPO eliminates the separate reward model step in RLHF.
• Write down the conceptual objective of the DPO loss function: $\mathcal{L}_{DPO}(\pi_\theta; \pi_{ref}) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w | x)}{\pi_{ref}(y_w | x)} - \beta \log \frac{\pi_\theta(y_l | x)}{\pi_{ref}(y_l | x)} \right) \right]$
• Understand what the hyperparameter $\beta$ controls (the scaling of the divergence from the reference policy).

---

Module 5: Data Preparation and the TRL Library

Curriculum Note: To address the lack of specialized data prep videos in the pool, this module combines the high-level project walkthroughs from available videos with a comprehensive Python guide showing you how to programmatically structure your preference data into the Hugging Face format.

Recommended Videos

Why this video: This clip provides context on how modern machine learning projects construct preference tuning workflows. It explains how multi-output generation works and how labeling preferred versus non-preferred completions shapes real-world datasets.

---

Why this video: This step-by-step pipeline uses TRL to ingest custom datasets. While it is primarily SFT-focused, it demonstrates the structural workflow of setting up datasets and feeds directly into the data preparation mindset needed for preference-based configurations.

---

Step-by-Step Dataset Prep Guide

The Hugging Face TRL library's DPOTrainer expects your dataset to contain exactly three columns representing a binary preference query:

1. prompt: The starting input system instruction and user query.
2. chosen: The high-quality or preferred model response.
3. rejected: The lower-quality, incorrect, or unaligned model response.

Python Code: Formatting Custom Raw Data

Here is a complete, production-ready script to convert raw Python dictionaries into a Hugging Face Dataset ready for your DPO trainer:

from datasets import Dataset

1. Define your raw feedback dataset

raw_feedback_data = [ { "prompt": "What is the capital of France?", "preferred_response": "The capital of France is Paris, located in the north-central part of the country.", "bad_response": "I think it is London. Or maybe Berlin, I forget." }, { "prompt": "Explain recursion simply.", "preferred_response": "Recursion is when a function calls itself to solve a smaller instance of the same problem, stopping at a base case.", "bad_response": "Recursion is just an infinite loop that crashes your computer eventually." } ]

2. Reformat the keys to match DPOTrainer specifications: 'prompt', 'chosen', 'rejected'

formatted_data = { "prompt": [], "chosen": [], "rejected": [] }

for item in raw_feedback_data: formatted_data["prompt"].append(item["prompt"]) formatted_data["chosen"].append(item["preferred_response"]) formatted_data["rejected"].append(item["bad_response"])

3. Create a Hugging Face Dataset object

dpo_dataset = Dataset.from_dict(formatted_data)

Print a preview to verify structural correctness

print("Dataset columns:", dpo_dataset.column_names) print("Sample item:", dpo_dataset[0])

Knowledge Checkpoint

• Create a custom preference dataset with at least three sample triplets containing your own prompts, chosen responses, and rejected responses.
• Convert your custom dataset into a Hugging Face Dataset object and print its schema using Python.
• Acknowledge why the DPOTrainer needs a rejected response column alongside a chosen column to compute model alignment loss.

---

Module 6: Implementing DPO: Fine-Tuning Code Walkthrough

Curriculum Note: To compensate for the limited step-by-step notebook tutorials in the video pool, this module provides a complete, runnable Python code template utilizing Hugging Face TRL's DPOTrainer alongside the conceptual demonstrations below.

Recommended Videos

Why this video: A practical step-by-step walkthrough showing how to initialize virtual environments, configure libraries, and construct an online DPO tuning script using the Hugging Face trl library.

---

Why this video: This clip highlights the system architecture advantages of running DPO over standard PPO models, validating why setting up this training loop mathematically scales better in practical, resource-constrained environments.

---

Step-by-Step DPOTrainer Implementation Script

Ensure you have the required packages installed in your notebook:

pip install transformers trl datasets accelerate peft bitsandbytes

Here is your complete implementation script. We use Low-Rank Adaptation (LoRA) to enable training on standard consumer GPUs:

import torch from datasets import Dataset from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments from trl import DPOTrainer from peft import LoraConfig

1. Prepare sample dataset (similar to Module 5)

data = { "prompt": [ "Human: Be polite and short. Tell me if you like coffee.\nAssistant:", "Human: Explain backpropagation in one sentence.\nAssistant:" ], "chosen": [ "Yes, I love coffee! It keeps my processes running fast.", "Backpropagation computes the gradient of the loss function with respect to the weights to update them." ], "rejected": [ "Coffee? Whatever, it's just a liquid.", "It is a complex math formula that goes backwards in neural nets." ] } dataset = Dataset.from_dict(data)

2. Select a base model and load its tokenizer

model_id = "facebook/opt-125m" # A small base model perfect for lightweight demonstration tokenizer = AutoTokenizer.from_pretrained(model_id) if tokenizer.pad_token is None: tokenizer.pad_token = tokenizer.eos_token

3. Load Policy Model and Reference Model

Note: DPOTrainer will automatically copy the base model to create the reference model

if we do not explicitly pass a separate ref_model.

model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.float16, device_map="auto" )

4. Define LoraConfig (Parameter-Efficient Fine-Tuning)

peft_config = LoraConfig( r=8, lora_alpha=16, target_modules=["q_proj", "v_proj"], lora_dropout=0.05, bias="none", task_type="CAUSAL_LM", )

5. Define Training Arguments

training_args = TrainingArguments( output_dir="./dpo_results", per_device_train_batch_size=1, gradient_accumulation_steps=4, learning_rate=5e-5, logging_steps=1, num_train_epochs=1, fp16=True, remove_unused_columns=False, # Essential for DPOTrainer to parse datasets natively )

6. Initialize DPOTrainer

dpo_trainer = DPOTrainer( model=model, args=training_args, beta=0.1, # The temperature parameter for DPO loss. Typical values range from 0.1 to 0.5. train_dataset=dataset, tokenizer=tokenizer, peft_config=peft_config, max_prompt_length=128, max_length=256, )

7. Execute DPO Alignment

print("Starting DPO training loop...") dpo_trainer.train() print("Training complete! Model aligned successfully.")

Knowledge Checkpoint

• Successfully run the code script above in your notebook environment.
• Explain why standard SFT parameters require remove_unused_columns=False when working with custom columns in the TRL framework.
• Tune the beta parameter to 0.5 and explain how a higher beta values-weighting penalizes deviations from the reference model.

---

Key People Index

• Umar Jamil — AI Researcher and Educator. Renowned for creating rigorous mathematical walk-throughs of foundational generative AI research papers (such as Transformers, RLHF, and DPO).
• Jeremy Howard — Co-founder of Fast.ai. Pioneer of accessible deep learning education. Advocated for high-level APIs that democratize model fine-tuning and state-of-the-art NLP training structures.
• Yann LeCun — Chief AI Scientist at Meta, Turing Award Winner. Known for his critiques of heavy reinforcement learning frameworks and pushing for efficient alternative self-supervised structures.

---

Final Self-Assessment

Complete this comprehensive final review to prove mastery over the entire LLM alignment pipeline:

• I can write a Python script from scratch using the datasets library to convert a flat file of text records into formatted pairs.
• I can explain the difference between Supervised Fine Tuning (SFT) and preference alignment.
• I can mathematically explain how DPO converts a reinforcement learning objective into a simpler binary cross-entropy classification loss.
• I can describe the roles of both the Active Policy Model ($\pi_\theta$) and the Static Reference Model ($\pi_{ref}$) during DPO.
• I can construct a structured dataset with prompt, chosen, and rejected keys.
• I can configure a LoraConfig object to point to target attention projections inside an LLM.
• I can initialize and execute a DPOTrainer session inside a Jupyter or Colab notebook.
• I can explain the function of the beta hyperparameter in DPO and identify the effects of raising or lowering it.
• I know how to check training loss curves to evaluate whether a model is successfully aligning to preference signals over epochs.