Reinforcement Learning with Human Feedback (RLHF) Explained

Large Language Models (LLMs) are AI systems that can converse about any topic, execute tasks, and create documents. They belong to Natural Language Processing (NLP), which studies human language. The Transformer architecture, proposed by Google Brain in 2017, revolutionized NLP by outperforming previous Recurrent Neural Networks (RNNs). Training requires massive datasets from CommonCrawl, a non-profit that crawls the entire internet. The data cleaning process involves: (1) RL filtering to remove sensitive content; (2) Text extraction to remove HTML tags; (3) Language filtering; (4) Privacy removal. Tokenization transforms text into tokens through UTF encoding, byte grouping, and Byte Pair Encoding (BPE), making processing more efficient. Transformers measure statistical relationships between token sequences to generate text by predicting the most likely next token at each step. LLMs generate text token by token through an autoregressive process: input text is converted to tokens, fed into the model, which outputs probability distributions over all possible next tokens, and a random selection picks the next token. Training involves billions of parameters (e.g., ChatGPT-3.5 has 175 billion) that are adjusted to improve prediction accuracy. A base model is a neural network trained on internet data that can generate text but lacks assistant capabilities. In-context learning allows models to understand patterns from examples provided in the prompt.

Large Language Models (LLMs) like GPT, Gemini, and Claude use transformer architecture to process language. The transformer converts text into high-dimensional vectors and uses matrix operations with billions of parameters to generate responses. The model generates text token-by-token by calculating probabilities based on previous tokens, similar to solving fill-in-the-blank questions. This process requires massive computational power, which is why companies are developing specialized AI chips.

LLM (Large Language Model) is an AI system that understands, manipulates, and generates human language. It is called 'large' due to billions of parameters trained on public datasets like websites and articles. The text generation process involves five steps: tokenization (converting text to tokens), embedding (converting tokens to vectors), transformation (pre-processing vectors), probability calculation (deciding next token), and sampling (selecting the token). Tokenization splits text into smaller units where common words become single tokens and long words are divided into multiple tokens. Different LLM models use different tokenizers with different token IDs for the same words.

LLMs generate text through an iterative token-by-token process where each token prediction is based on probability distributions learned during training. Tokenization converts raw text into numerical tokens (words, subwords, or characters) assigned unique numerical IDs, as LLMs only understand numerical data. Tokens are converted into vector representations that capture meaning and context, allowing the model to understand that the same word can have different meanings in different contexts. The complete LLM architecture consists of three key components: pre-training (learning from data), transformers (enabling context understanding), and generation (creating responses). Generative AI encompasses text-to-text, text-to-audio, text-to-video, and text-to-image generation, all working through the same token-by-token prediction process.

This section covers the Transformer architecture and Large Language Models (LLMs). The Transformer consists of input embedding, positional encoding, multi-headed attention, normalization, and feed forward networks, with both encoder and decoder segments. This enables parallel processing, making it faster than RNN-based models. LLMs are trained on huge amounts of data through unsupervised learning (learning patterns), supervised fine-tuning (learning instructions), and reinforcement learning (optimizing for helpfulness). Key milestones include BERT (encoder-only), GPT family (decoder-only), XLM (cross-lingual), T5 (encoder-decoder), and Megatron. LLMs can perform multiple tasks including text generation, chatbot creation, summarization, translation, and code generation. They are categorized by Transformer architecture: encoder-only models (BERT, RoBERTa, XLM, ALBERT, ELECTRA), decoder-only models (GPT family), and encoder-decoder models (T5, BART, M2M-100). OpenAI provides models like GPT-4, GPT-3.5, and GPT-3.5 Turbo. Open source alternatives include Bloom, Llama 2, Falcon, and StableLM. Hugging Face provides a Models Hub with various open source LLMs. LLMs enable transfer learning in NLP by being trained on huge amounts of data, allowing them to learn general patterns that can be fine-tuned for specific tasks like sentiment analysis, named entity recognition, or language translation.

Supervised Fine-Tuning (SFT) is a training technique that transforms raw language models into helpful assistants by teaching them to follow instructions, using the same algorithm as pre-training but with a loss mask that only grades response tokens, requiring only 1,000-10,000 examples to achieve significant quality improvements, with quality plateauing early at around 85% and demonstrating that knowledge comes from pre-training while behavior comes from SFT.

Supervised Fine-Tuning (SFT) transforms raw pre-trained language models into helpful instruction-following assistants by training them on curated prompt-response pairs using loss masking, where the model learns to ignore its own prompts and only optimize for generating expert responses; however, SFT has a fundamental limitation of treating all good answers equally well without understanding quality differences, which requires additional preference-based training methods to address.

Supervised Fine-Tuning (SFT) transforms pre-trained models into instruction-following models by training on instruction-response pairs. The model learns to predict response tokens following instructions, enabling tasks like writing essays and preparing presentations. SFT differs from pre-training where models simply predict next tokens. The process involves creating datasets with natural language instructions and desired responses, then training the model to map instructions to appropriate outputs. This transforms models from passive next-token predictors to active instruction-following assistants.

Supervised Fine-Tuning (SFT) is the post-training technique used to align pre-trained language models into useful AI assistants. The process involves fine-tuning the pre-trained model on human-labeled question-answer pairs, where humans provide desired responses to specific prompts. SFT teaches the model to follow instructions and produce helpful answers rather than just modeling internet text patterns. Despite requiring relatively little data (around 2,000-32,000 examples), SFT dramatically improves practical usefulness by specializing the general-purpose language model toward helpful assistant behavior.

Supervised Fine-Tuning (SFT) transforms pre-trained LLMs into instruction-following assistants by retraining on instruction-answer datasets. The goal is not to add new knowledge but to adapt models to specific tasks like document generation or PDF explanation. SFT works best when leveraging existing knowledge in the base model; attempting to teach completely new domains leads to hallucinations. Three main techniques exist: Full Fine-Tuning (retraining all parameters, impractical due to resource requirements and catastrophic forgetting), LoRA (Low-Rank Adaptation, freezes weights and adds small adapter matrices for <1% trainable parameters), and QLoRA (Quantization-Aware LoRA, provides additional 33% memory reduction). Unsloth library offers 2x faster training and 60% memory reduction for Llama models. Essential libraries include Unsloth, Transformers, TRL, and PyTorch. Model loading uses FastLanguageModel.from_pretrained with specific model names and parameters like max_sequence_length (2048 tokens) and load_in_4bit=True for memory efficiency.

Proximal Policy Optimization (PPO) is a policy gradient method designed at OpenAI that balances sample efficiency, ease of implementation, and tuning simplicity. Unlike traditional policy gradients that suffer from unstable training due to changing data distributions and hyperparameter sensitivity, PPO addresses these issues by using a clipped probability ratio objective function that limits how far the new policy can diverge from the old policy. The algorithm combines the clipped PPO objective with value function estimation and entropy regularization in a unified loss function, achieving state-of-the-art performance across diverse tasks including robotic control, Atari games, and complex multi-agent environments like Dota 2.

Proximal Policy Optimization (PPO) is a reinforcement learning algorithm introduced by OpenAI in 2015 that improves training stability by limiting the magnitude of policy updates through a clipped surrogate objective function, which constrains the ratio between the new and old policy probabilities to stay within [0.8, 1.2], thereby preventing excessive greedy updates that could destabilize learning.

Proximal Policy Optimization (PPO) is a reinforcement learning algorithm that simultaneously trains two neural networks—a value network and a policy network—to approximate the optimal value function and policy for an agent. The value network is trained using a squared loss function that minimizes the difference between predicted and actual values, while the policy network uses a clipped surrogate objective function that incorporates momentum through the ratio of current to previous action probabilities, ensuring stable and efficient training of large language models through reinforcement learning with human feedback.

Policy gradient methods directly optimize the policy parameters by computing gradients of expected returns. Traditional policy gradients suffer from instability due to large parameter updates that can drastically alter the policy. Trust Region Policy Optimization (TRPO) addressed this by constraining updates but introduced implementation challenges. Proximal Policy Optimization (PPO) improves upon TRPO by using a surrogate loss function with clipping mechanisms that limit policy changes per update, ensuring smooth and stable learning progress. PPO represents a significant advancement in policy gradient methods for practical RL applications.