How to Fine-Tune Llama 3 on Custom Data with Unsloth

Traditional metrics like BLEU, ROUGE, and F1 only measure surface-level features such as word overlap and binary correctness, failing to capture deeper qualities like factual accuracy, helpfulness, safety, and domain-specific knowledge in large language models; modern evaluation approaches address this gap through LLM-as-a-Judge methods where one LLM evaluates another using pairwise comparisons or rubric-based scoring, and through LLM-generated benchmarks that create adaptive, domain-specific tests for real-world accuracy assessment.

LLM-as-a-judge is an evaluation method where large language models assess each other's outputs by classifying or scoring text against specific criteria (such as politeness, conciseness, or factual accuracy), serving as a scalable alternative to human evaluation for monitoring and improving LLM performance in production environments.

Model evaluation determines how smart and accurate a model is. Basic evaluation involves grading responses against known correct answers using benchmarks like MMLU. For coding tasks, execution-based benchmarks like SW bench simulate real software engineering work. For open-ended responses without single correct answers, reference-free evaluation using LLM-as-a-Judge becomes essential. This method combines user input and model response, sends it to an evaluating LLM with a trained rubric, and requires human experts to validate scores. This approach enables scalable evaluation of hundreds or thousands of outputs while maintaining quality control.

LLM-as-a-judge evaluation is a method for assessing AI systems by using large language models as automated judges that evaluate outputs based on predefined criteria, following a four-step process: (1) analyze your data to understand failure modes, (2) define specific, actionable metrics, (3) write clear prompts and select appropriate judge models, and (4) perform meta-evaluation by comparing LLM judgments against human annotations to ensure reliability; effective evaluations require starting with good data, keeping metrics specific and answerable rather than vague or overly numerous, limiting context windows to around 4,000 tokens, and recognizing that LLM judges should complement rather than completely replace human judgment.

The LLM-as-a-Judge paradigm uses Large Language Models to evaluate other LLM responses, providing scalable automated quality assessment. This addresses risks of human evaluation being time-consuming and potentially biased. The system uses a committee of LLMs to score responses based on context relevance, answer relevance, and faithfulness. Different models show different strengths: Llama 3.1 excels in relevance, Llama 3.2 in faithfulness, and GPT-4o provides the best balance.

vLLM does not support GGUF quantization format (used by Ollama), requiring alternative formats like AWQ. AWQ is the recommended quantization format for vLLM deployments, supporting both AWQ and GPTQ approaches. While GGUF offers multiple quantization levels, performance typically degrades significantly below 4-bit quantization. AWQ provides a practical trade-off, offering good performance without requiring deep expertise in quantization formats. Users can find AWQ versions of models on Hugging Face for deployment.

Model quantization is a technique that reduces computational and memory requirements of deep learning models by representing model parameters using fewer bits, typically converting high-precision weights and activations into lower-precision values (such as 8-bit integers or even fewer bits) while minimizing the impact on model accuracy. The GGUF format is a binary file format used to store and load model weights for the llama.cpp library, enabling efficient CPU-based inference. AutoQuant is a tool that enables users to quantize models in multiple formats including GGUF, GPTQ, EXL2, AWQ, and HQQ, with GGUF being accessible on commodity hardware like Google Colab's T4 GPU.

vLLM is an open-source Apache 2.0 library for efficient large language model inference, supporting diverse architectures (Llama 3.1, vision-language models) and hardware (NVIDIA, AMD, Intel GPUs, TPUs). Recent releases include expanded model support, pipeline parallelism for Vision Language Models, soft tune prompts, hardware support for AMD and TPUs, low adapter loading from Hugging Face, an AWQ Marlin kernel, FP8 Cutlass kernel optimizations, and CPU offloading. Model quantization reduces neural network precision to improve inference efficiency through four key benefits: reduced latency, lower deployment costs, decreased GPU resource requirements, and reduced energy consumption. Two critical questions guide quantization decisions: whether deploying for online (low-latency, memory-bound) or offline (high-throughput, compute-bound) inference, and which GPU architecture is available. For online scenarios, weight-only quantization is optimal; for offline, weight-and-activation quantization leverages Turing/Hopper tensor cores for 2x speedup. LM Compressor addresses format fragmentation by enabling quantization of any Hugging Face model into vLLM-compatible checkpoints, supporting GPTQ, AWQ, SmoothQuant, and pruning algorithms with evaluation via Llama Evaluation Harness.

Model quantization reduces numerical precision from 32/16-bit to 4-bit representations, enabling LLMs to run on smaller devices and GPUs. Key benefits include reduced VRAM requirements (e.g., Llama 70B from 2-3 A100 GPUs to single A6000 or Mac M1), lower costs, and broader accessibility. Four major approaches differ significantly: (1) Data dependence—AWQ/GPTQ require training data for adaptive quantization, while GGUF/BitsAndBytes are data-independent; (2) On-the-fly support—BitsAndBytes enables dynamic quantization during inference, while data-dependent methods require pre-processing; (3) Speed—GPTQ with act_order is slower than AWQ; (4) Fine-tuning support—BitsAndBytes/GPTQ/GGUF support LoRA, but AWQ lacks this capability; (5) Model sharing—GGUF/AWQ save directly to Hugging Face, while BitsAndBytes requires merging first. Selection depends on hardware (GGUF for laptops, AWQ for GPUs) and use case (fine-tuning vs. inference).

GGML (Georgi Gerganov's Machine Learning library) is a C-based inference engine for efficient LLM inference on CPU and Apple Silicon. GGUF (Gorgor Unified Format) is a self-describing binary file format storing quantized models with metadata. GGUF is the successor to GGML's binary format, providing better portability, versioning, and tokenizer support. llama.cpp is a C++ extension of GGML providing enhanced inference capabilities. The workflow involves: quantization (GPTQ/AWQ), conversion to GGUF, and inference using llama.cpp.

This tutorial demonstrates a systematic approach to curating high-quality text datasets for fine-tuning large language models (LLMs), involving data filtering and cleaning (removing HTML tags, normalizing Unicode, applying length constraints), semantic deduplication to remove redundant records, and synthetic data generation using prompt templates to expand dataset size while maintaining quality through reward model scoring.

This section explores advanced data curation strategies and their practical implications for enterprise AI deployment. Synthetic data generation has two fundamentally different approaches: net-new creation risks model collapse where repeated generation creates increasingly narrow distributions, while rephrasing leverages information already present in original data, avoiding these risks. Research demonstrated that rephrasing can produce models better than either the teacher model generating data or the base model being trained. Curriculum learning - ordering data to teach concepts progressively - had fallen out of favor in supervised learning regimes where data was fully saturated, but in the current underfitting regime where models are never truly saturated, curriculum becomes critically important. Research conceptualized learning as a graph where nodes represent concepts and edges represent dependencies between them; curricula only make sense when this graph is neither empty nor complete. Enterprises pursuing AI deployment prioritize different aspects based on maturity: train faster matters for iteration speed, train better acts as a compute multiplier, and train smaller becomes critical for inference cost optimization. Model pruning has limitations that make it complementary rather than competitive with data-driven approaches, as pruning effectiveness is highly data-dependent.

Instruction tuning (supervised fine-tuning) is the first step in post-training where models learn to follow instructions. Data curation is critical: human annotation provides high-quality examples but is expensive and has high variance; synthetic generation using self-instruction loops is scalable but introduces noise. The optimal approach combines both data sources. Creating effective instruction-tuned models requires mixing multiple datasets while maintaining performance across diverse capabilities—different datasets excel at different tasks. Chain of Thought (CoT) data is essential for reasoning by showing step-by-step problem-solving processes. Researchers developed persona-driven synthetic generation, assigning different personas to generate diverse problems, and improved quality through self-consistency filtering, reducing dataset size by 40% while maintaining performance on challenging benchmarks.

An effective approach to golden dataset creation combines synthetic generation with human validation. Organizations provide diverse examples of expected query types, iterate over knowledge bases to generate examples, and extract answers from documents. This speeds up the workflow while catching hallucinations that need correction. Starting with cleaner data reduces mismatches, making the golden dataset creation process more efficient and resulting in higher-quality benchmarks.

Creating effective training data requires understanding the relationship between data format and learning objective. Supervised fine-tuning needs instruction-response pairs (questions with answers). Preference-based training requires instruction, winning response, and losing response to teach distinction between good and bad outputs. Reward modeling uses scored examples to teach quality assessment. Beyond format, the critical challenge is ensuring the model learns the intended behavior—for reasoning tasks, models need explicit chains of thought showing complete logical steps, not just final answers. For conversational agents, multi-turn dialogue data teaches context maintenance. Crucially, curation involves removing problematic examples that induce unwanted behaviors like arrogance or intrusiveness. This is artisanal work requiring human judgment, where small improvements yield significant gains.