Fine-Tuning LLMs: LoRA, PEFT & Custom Data

Learning Goal: Fine-tune open-source Large Language Models (LLMs) using Parameter-Efficient Fine-Tuning (PEFT) and LoRA for domain-specific text generation.

Prerequisites

• Advanced Python: Mastery of object-oriented programming (OOP), decorators, context managers, and standard scientific libraries (NumPy, Pandas).
• Basic Linear Algebra: Solid understanding of vectors, matrices, matrix multiplication, and matrix dimensionality.
• Hardware Requirements: Access to a computer with a modern NVIDIA GPU (preferably with 16GB+ VRAM) or an active Google Colab account (with T4 or A100 GPU runtimes).

Estimated Study Time

• Total Duration: 24 Hours (including self-guided programming exercises and fine-tuning runs).

---

Module 1: PyTorch Tensors & Deep Learning Basics

This module focuses on the core mechanics of deep learning and PyTorch, bypassing high-level Python syntax to dive directly into the mathematical engine powering modern neural network architectures. You will learn how tensors store spatial information, how weights and biases transform inputs, and how PyTorch's automatic differentiation engine (autograd) manages backpropagation.

• Why this video: This video provides an unmatched visual intuition of neural network topologies. It visually breaks down the concept of mathematical layers, weights, and biases, translating the abstract linear algebra of forward propagation into a physical, understandable representation.

• Why this video: A rapid, high-level structural overview of PyTorch. It explains how PyTorch combines standard dynamic computational graphs with strong GPU processing speeds to replace older frameworks, explaining its massive popularity in academic research and industry.

• Why this video: This hands-on crash course bridges theory and code, walking you through fundamental PyTorch operations. You will learn how to initialize and manipulate multi-dimensional tensors, utilize GPU acceleration, and construct customizable modular neural layers by subclassing torch.nn.Module.

Knowledge Checkpoint

• Implement a custom multi-layer neural network class that inherits from torch.nn.Module.
• Explain the difference between tensor operations executed on a CPU vs. those offloaded to an NVIDIA GPU via .to('cuda').
• Walk through how the dynamic computational graph in autograd tracks gradients during forward and backward passes.
• Mathematically explain how backpropagation updates a simple weight using the chain rule.

---

Module 2: NLP Foundations & The Transformer Architecture

This module traces the evolution of natural language processing from simple heuristic text parsers to modern transformer neural networks. You will explore how written characters are translated into high-dimensional semantic vector spaces, track positional order, and dissect the self-attention mechanism that allows models to capture long-range contextual relationships.

• Why this video: Provides a clear foundational view of historical and modern NLP pipelines. It outlines how unstructured natural language is transformed into structured inputs that machine learning models can ingest and evaluate.

• Why this video: Famous for breaking down complex concepts, StatQuest provides an extremely clear, step-by-step mathematical breakdown of the Transformer encoder-decoder architecture. It clearly illustrates word embeddings, positional encodings, and attention calculation.

• Why this video: Delivered by Andrej Karpathy at Stanford, this lecture is a masterclass on the historical pivot from RNNs/LSTMs to self-attention. Karpathy breaks down the physical intuition of query, key, and value vectors, explaining how attention scales across modern GPUs.

Knowledge Checkpoint

• Describe why Recurrent Neural Networks (RNNs) struggle with long sequences, and how the self-attention mechanism resolves this bottleneck.
• Calculate the scale-dot product attention score given small sample Query (Q), Key (K), and Value (V) matrices.
• Define the role of positional encoding and explain why transformers require it to preserve sequence order.
• Differentiate between encoder-only, decoder-only, and encoder-decoder transformer architectures.

---

Module 3: Working with LLMs & Hugging Face

This module covers the core ecosystem of open-source artificial intelligence: Hugging Face. You will learn the mechanics of tokenization, handle special tokens, and explore how to use the high-level Hugging Face API to load state-of-the-art models for inference.

• Why this video: This video (featuring Andrej Karpathy's introductory lecture) provides a solid overview of LLM training phases: pre-training (compressing the internet into parameters) and post-training (instruction fine-tuning and safety alignment).

• Why this video: A practical, code-focused guide to Hugging Face pipelines. This video shows how to instantiate pre-trained architectures with just a few lines of code to perform diverse downstream language tasks.

• Why this video: This tutorial demonstrates how to load models programmatically using AutoModelForCausalLM and AutoTokenizer, and walks you through creating a simple web-based prototype chatbot interface using Gradio.

Knowledge Checkpoint

• Programmatically tokenize a raw text string and identify standard token components (such as attention masks, input IDs, and special start/end tokens).
• Differentiate between pre-training an LLM (next-token prediction on raw text corpora) and Supervised Fine-Tuning (SFT).
• Instantiate a pre-trained open-source causal language model using PyTorch and Hugging Face transformers.
• Write a script that runs inference locally, demonstrating basic token decoding to convert numerical tensors back into natural language.

---

Module 4: Fine-Tuning Concepts & PEFT/LoRA Theory

This module explores parameter-efficient tuning methods, comparing full-parameter training with adapter-based updates. You will study the mathematics of Low-Rank Adaptation (LoRA), analyzing how decomposing parameter weight updates into lower-rank matrices reduces GPU memory overhead.

• Why this video: Keerti Purswani delivers a clear, high-level conceptual overview comparing standard full-parameter fine-tuning with PEFT alternatives like LoRA and QLoRA, highlighting their memory-saving benefits.

• Why this video: An in-depth academic lecture that covers the mathematical foundations of PEFT. It dives into parameter freezing, structural parameter manipulation, and the performance trade-offs of modern adapter layers.

• Why this video: This video provides a detailed structural breakdown of QLoRA. It covers the four central pillars of QLoRA: 4-bit NormalFloat (NF4) quantization, Double Quantization, Page Optimizers, and the execution of LoRA adapters over frozen base model weights.

Knowledge Checkpoint

• Explain the LoRA equation $W_{new} = W_0 + \Delta W$, where $\Delta W = B \times A$, and describe how this low-rank decomposition saves memory.
• Mathematically calculate the reduction in trainable parameters when applying a LoRA adapter with rank $r=8$ to a $4096 \times 4096$ weight projection matrix.
• Define the hyperparameter $\alpha$ (alpha) in LoRA configuration and explain its role in scaling adapter weight contributions.
• Explain the memory savings achieved by combining 4-bit NormalFloat base-weight quantization with active, high-precision adapter matrices.

---

Module 5: Hands-On Dataset Formatting & LoRA Fine-Tuning

This module guides you through the practical, hands-on steps of preparing and fine-tuning models. You will learn how to structure raw text datasets into instruction-tuning formats (such as Alpaca, ChatML, or Llama chat templates) and write PyTorch scripts using the Hugging Face TRL (Transformer Reinforcement Learning) library to train a custom model.

• Why this video: A clear step-by-step walkthrough on how datasets must be structured for supervised fine-tuning. Abhishek Thakur demonstrates how to map raw data fields into the precise query-response sequences required by Hugging Face formatting utilities.

• Why this video: This tutorial walks through setting up a complete QLoRA fine-tuning pipeline on a single GPU. It covers loading custom JSON data, configuring parameter-efficient matrices, and executing the training loop using the Hugging Face SFTTrainer.

• Why this video: Focuses on the modern Unsloth framework for fine-tuning newer models like Llama 3.2. It demonstrates how to format instruction datasets (using templates like FineTome-100K) and run optimized, high-speed training on free Google Colab hardware.

Knowledge Checkpoint

• Format a raw dataset into a valid training schema (e.g., Alpaca or ChatML) using Hugging Face's Dataset.map() operations.
• Configure LoraConfig from the peft library, explaining your choices for rank ($r$), target modules, and $\alpha$.
• Instantiate and execute an SFTTrainer (Supervised Fine-Tuning Trainer) with active gradient checkpointing and mixed-precision (FP16 or BF16) enabled.
• Write a script that catches Out-of-Memory (OOM) exceptions and optimizes GPU batching configurations using gradient accumulation.

---

Module 6: LLM Evaluation, Quantization & Deployment

Once a model is fine-tuned, you need to validate its performance and prepare it for production. This module covers performance assessment (using validation loss curves, generation tests, and benchmark metrics), advanced post-training quantization techniques, and local serving using Ollama.

• Why this video: Although brief, this video highlights why evaluation suites and benchmarks are essential for modern AI engineering. It explains that qualitative testing is not enough; you must build quantitative evaluation pipelines to prove your fine-tuned model outperforms the baseline.

• Why this video: This video covers the post-training deployment pipeline, walking you through saving fine-tuned models, converting weights to GGUF format, and pushing customized templates to Ollama for local hosting.

• Why this video: A comprehensive, end-to-end deployment guide. TechWithTim walks through exporting trained LoRA adapters, merging those adapters back into baseline base model models, and configuring local model environments with customized system prompts.

Knowledge Checkpoint

• Evaluate model convergence using training and validation loss curves, monitoring for overfitting.
• Run qualitative evaluations on your model by generating text across multiple prompt test suites to check for performance regression.
• Merge trained LoRA weights back into baseline model weights using .merge_and_unload().
• Export your fine-tuned model into GGUF format and instantiate a local model API instance via Ollama.

---

Course Map

This flowchart maps out the curriculum modules and recommended learning progression.

---

Key People Index

• Andrej Karpathy: Renowned AI educator, former Tesla Autopilot Director, and OpenAI co-founder. His instructional videos (covered in Modules 2 & 3) are industry-standard references for understanding core LLM concepts.
• Grant Sanderson (@3blue1brown): Creator of the 3Blue1Brown math channel. His visualizations of neural network weights, biases, and transformations (covered in Module 1) provide the conceptual foundations for modern machine learning.
• Tim Dettmers: Leading researcher behind bitsandbytes and the primary author of the QLoRA paper. His work on 4-bit NormalFloat and double quantization made single-GPU fine-tuning accessible to the open-source community.

---

Final Self-Assessment

Test your practical and theoretical understanding of the concepts covered in this curriculum:

• Linear Algebra Basis: I can visually and mathematically explain why updating weights using a low-rank adapter matrix decomposition ($B \times A$) requires significantly less GPU memory than updating a full-rank weight matrix ($W$).
• Autograd Tracking: I understand when PyTorch builds computational graphs and how to temporarily disable memory tracking using torch.no_grad() during model evaluation and inference.
• Self-Attention Mechanics: I can explain how the Self-Attention mechanism uses Query, Key, and Value matrices to calculate token relationships across sequence inputs.
• Subclassing Models: I can build, train, and save custom neural network layers using PyTorch's native nn.Module and optimization tools.
• Tokenization & Formatting: I can write a Python script using Hugging Face APIs to ingest raw JSON datasets and map them into structured, padded instruction-tuning templates (like ChatML).
• Hyperparameter Configuration: I can confidently configure LoRA adapter rank ($r$), alpha scaling ($\alpha$), target modules, and dropout ratios based on hardware constraints and task complexity.
• Quantization Mechanics: I can explain the difference between uniform integer quantization and 4-bit NormalFloat (NF4) quantization.
• Model Merging: I can merge fine-tuned LoRA adapter weights back into a frozen base model to produce unified FP16/BF16 weights.
• Local Inference Serving: I can export a merged model to GGUF format and run a local inference API endpoint using Ollama.