Academic Databases
Explore Our Semantic Databases
STEM & Tech
Math, Physics, CS & Engineering...
Economics
Macro, Micro, Econometrics...
Business
Management, Marketing, Finance...
Health & Life Sciences
Biology, Medicine, Neuroscience...
Humanities & Social Sciences
Psychology, Law, Philosophy, History...
General Studies
General & interdisciplinary studies...
Explore Topics
Features

Fine-Tuning Llama 2: Custom Dataset LLM Training with QLoRA

Added: 2026-08-06

Deep Dive

Fundamental understanding of the Transformer architecture, particularly decoder-only language models like Llama.

Decoder-only inference: a step-by-step deep dive

Decoder-only inference: a step-by-step deep dive

31.7K

Text generation models use decoder-only architectures, which differ from the original Transformer architecture that included both encoder and decoder components. The decoding process occurs in two main steps: (1) Input processing where the prompt is tokenized, converted to embeddings with positional encoding, and attention calculations are performed to compute important K (keys) and V (values) matrices; (2) Text generation where new tokens are generated sequentially, one at a time. Each newly generated token becomes part of the input sequence and is used to generate the next token, continuing until an end-of-sequence token is reached or the maximum number of tokens is generated.

Click to watch →

George Hotz | Programming | can we fit a LLaMA inside a tinygrad? | $1499 comma.ai/shop/comma-three

George Hotz | Programming | can we fit a LLaMA inside a tinygrad? | $1499 comma.ai/shop/comma-three

34.8K

Llama is a large language model released by Facebook (Meta) as an open-source project, contrasting with Google's closed-source approach. The model architecture is transformer-based, using attention mechanisms and feed-forward networks. The speaker explains that Llama uses rotary position encoding (RoPE) for positional information and is unbiased (no biases in model weights). Running Llama requires significant memory resources, with the model needing over 32 gigabytes when running on CPU, making GPU execution more feasible. The model uses float16 data type for weights, which is half the size of float32, reducing memory requirements. Transformer blocks are the fundamental building blocks, each containing an attention mechanism and a feed-forward network. The model uses RMSNorm (Root Mean Square Layer Normalization) for normalization, which computes the root mean square of the input and normalizes by this value, then applies a learned scaling factor. The speaker demonstrates how to implement RMSNorm in TinyGrad, showing how to compute the RMS of the input, divide by this value, and multiply by a learned weight.

Click to watch →

Карина Романова - LLAMA, 100500 вариантов её файнтюнинга

Карина Романова - LLAMA, 100500 вариантов её файнтюнинга

15.6K

Llama is Meta's large language model designed to help researchers advance AI work, notable for being smaller than GPT-3 while achieving competitive performance. Built on the Transformer architecture from 2017, it incorporates key innovations: layer normalization (dividing input vectors by mean squared elements), Swish activation (smoother than ReLU with better generalization), and Rotary Position Embeddings (RoPE) encoding both absolute and relative token positions. Training optimization techniques include weight decay regularization, AdamW optimizer, gradient clipping at 1.0, and linear complexity optimization for multi-head attention. Activation checkpointing reduces memory usage by recomputing intermediate activations. BPE tokenization compresses text by merging frequent byte pairs. These techniques enable efficient training of large models on limited hardware.

Click to watch →

Llama 3 and DeepSeek V3 Architecture & Applications | Lecture 15 | LLM 2025

Llama 3 and DeepSeek V3 Architecture & Applications | Lecture 15 | LLM 2025

322

The Transformer architecture consists of tokenization, embedding layers, and stacked Transformer blocks. Tokenization converts text to token IDs using a vocabulary of 128,000 tokens in Llama 3. Embedding combines token embeddings with positional encodings—Llama 3 uses rotational positional encoding (RoPE) instead of additive methods. The decoding phase uses 32 stacked Transformer blocks containing self-attention and feed-forward networks. Auto-regressive decoding generates one token at a time, with each new token fed back as input for the next generation step.

Click to watch →

The Truth about ChatGPT and Friends — understand what it really does and what that means

The Truth about ChatGPT and Friends — understand what it really does and what that means

7K

This section covers the transformer architecture that revolutionized LLMs. The attention mechanism (invented in the 1990s) allows decoders to access encoder intermediate states for context. The transformer solved RNN limitations by making all runs independent, enabling massive parallelism during training and allowing extremely large models. GPT is decoder-only—it doesn't encode input but generates output sequentially. The process: user provides prompt, model predicts next word, appends it to prompt, repeats until <END> token. Each prediction considers the entire sequence so far, with attention calculating relationships between all tokens to determine the most likely next word from a 100,000-word vocabulary.

Click to watch →

Deep Dive

The concept of Parameter-Efficient Fine-Tuning (PEFT) and why it is preferred over full parameter fine-tuning under resource constraints.

Fine-Tuning

Fine-Tuning

237

Parameter-Efficient Fine-tuning (PEFT) reduces computational cost by training only a small fraction of model parameters. Full fine-tuning updates all parameters (billions), while PEFT methods like LoRA add small adapter layers. LoRA decomposes weight matrices into low-rank matrices, reducing trainable parameters from 20,000 to approximately 300. This achieves similar results to full fine-tuning with significantly lower computational requirements. PEFT is particularly valuable for smaller models or when computational resources are limited. The choice between PEFT and full fine-tuning depends on the specific task requirements and available resources.

Click to watch →

LLMs | Parameter Efficient Fine-Tuning (PEFT) | Lec 14.1

LLMs | Parameter Efficient Fine-Tuning (PEFT) | Lec 14.1

4.8K

Parameter Efficient Fine-Tuning (PEFT) represents a paradigm shift addressing the trade-off between in-context learning and full fine-tuning. The core insight is to freeze most model parameters while training only a small subset, retaining benefits of both approaches: avoiding hardware/storage requirements of full fine-tuning while achieving better performance than manual prompting. Key advantages include: reduced hardware needs (only 12-20x trainable parameters instead of full model size), faster convergence, reduced overfitting risk, prevention of catastrophic forgetting (maintaining world knowledge when learning new tasks), and dramatically reduced storage requirements. This approach democratizes LLM customization by enabling effective fine-tuning on modest infrastructure.

Click to watch →

Fine-tuning de grandes modelos de lenguaje con Manuel Romero | Hackathon Somos NLP 2023

Fine-tuning de grandes modelos de lenguaje con Manuel Romero | Hackathon Somos NLP 2023

7K

PEFT addresses the high computational cost of fine-tuning large models. While large models can solve tasks with few examples, full fine-tuning remains prohibitively expensive. PEFT methods adjust only a small portion of parameters, achieving results comparable to full fine-tuning while significantly reducing computational costs. This enables researchers to leverage large-scale models without prohibitive expenses.

Click to watch →

Parameter Efficient Fine-Tuning Explained | PEFT for LLMs

Parameter Efficient Fine-Tuning Explained | PEFT for LLMs

9.3K

Parameter Efficient Fine Tuning (PEFT) is a technique that reduces the number of trainable parameters during the fine-tuning phase of large language models by introducing small adapter layers (multi-layer perceptrons with bottleneck architecture) between Transformer layers, thereby reducing memory usage and computational cost while achieving performance comparable to full fine-tuning; for example, in BERT-large, PEFT reduces trainable parameters from 345 million to approximately 6.34 million (1.8% of total parameters) per task.

Click to watch →

LLMs Fine-Tuning تطبيق عملي على ال

LLMs Fine-Tuning تطبيق عملي على ال

26.2K

Matrix multiplication is fundamental to neural networks, where matrices with dimensions (rows × columns) must have matching inner dimensions for multiplication. PEFT addresses the high memory requirements of traditional fine-tuning by creating small adapter networks that are trained separately from the frozen base model. The input is processed by the base model, then passed through the adapter network, with outputs combined for the final result. This approach allows fine-tuning on consumer hardware because adapter networks are much smaller than full models, democratizing access to advanced LLM customization techniques.

Click to watch →

Deep Dive

Basic knowledge of model quantization principles, including how reducing weight precision (e.g., to 4-bit) saves GPU memory.

Frontier AI On Your Laptop.

Frontier AI On Your Laptop.

569

Quantization reduces model memory requirements by decreasing the precision of weight values. At 16-bit precision, each weight can take 65,000 values; at 4-bit, only 16 values; at 2-bit, just 4 values. This can shrink a 568 GB model to approximately 80 GB. However, precision is not free—when important weights are quantized to incorrect levels, errors compound through the deep transformer layers. Models typically maintain quality down to 4-bit quantization but experience significant quality degradation below 3-bit quantization. This creates a dilemma: quantization saves memory but may sacrifice model intelligence.

Click to watch →

How We Cut GPU Costs 10x — 5 Tricks ($60K → $6K) #shorts

How We Cut GPU Costs 10x — 5 Tricks ($60K → $6K) #shorts

1.1K

Quantization is a technique that reduces the precision of model weights from 16-bit to 4-bit representation. This compression approach halves the GPU memory requirements for running large language models. By reducing the numerical precision while maintaining acceptable model accuracy, quantization enables more efficient inference on limited hardware resources.

Click to watch →

The Bizarre True Story of OpenAI’s Autonomous Worm Hack

The Bizarre True Story of OpenAI’s Autonomous Worm Hack

160

Model quantization is the process of reducing the precision of model weights from 16-bit to lower bit depths (such as 2-bit or 4-bit) to reduce memory requirements. This is analogous to reducing audio quality from 16-bit to 8-bit or 6-bit, where lower precision results in some loss of accuracy but enables the model to run on devices with limited memory.

Click to watch →

Napkin Math For Fine Tuning Pt. 1 w/Johno Whitaker

Napkin Math For Fine Tuning Pt. 1 w/Johno Whitaker

3.6K

Quantization reduces memory usage by storing model weights with fewer bits per parameter. Common approaches include 8-bit (quarter the size of 32-bit) and 4-bit (eighth the size) representations. For a large model taking up 60% of GPU memory in 32-bit format, quantization can reduce this to 5-8% of GPU capacity. This enables training larger models or fitting models that previously exceeded available memory, though extreme compression (below 4 bits) may cause performance degradation.

Click to watch →

AI Quantization Easily Explained

AI Quantization Easily Explained

491

Quantization progressively reduces GPU memory requirements: 32-bit models require full capacity, 16-bit quantization halves memory needs (from 8GB to 4GB for a 7B model), and 8-bit quantization quarters memory needs (from 8GB to 2GB). This enables running larger models on consumer hardware.

Click to watch →

Deep Dive

Familiarity with the Hugging Face library ecosystem (Transformers, Datasets, and PEFT) and basic PyTorch.

Hugging Face : La bibliothèque IA qui change tout !

Hugging Face : La bibliothèque IA qui change tout !

10.6K

Hugging Face offers various assistants for AI tasks including image generation, video generation, audio generation, music generation, code generation, and document analysis. Spaces functions as an App Store for AI applications, allowing users to search for specific tasks (face swap, image editing, video generation) and find corresponding models. Some Spaces are marked as 'running on zero,' providing free access without GPU consumption. Transformers is a library grouping numerous Large Language Models for specific tasks, with Python code provided for local integration. Datasets provide databases for training machine learning models, including time series, text, video, and audio data, filterable by format (CSV, parquet) and data type. The platform offers both free and Pro versions (9 dollars/month) with community documentation and support.

Click to watch →

Matt Squire - Diving into Transformer Model Internals | PyData London 25

Matt Squire - Diving into Transformer Model Internals | PyData London 25

3.3K

The Hugging Face transformers library originated as a PyTorch reimplementation of Google's BERT model (originally TensorFlow) and expanded to support multiple frameworks (PyTorch, TensorFlow, Jax) and numerous models. The library structure includes AutoTokenizer for automatic tokenizer selection based on model, AutoModel for task-specific model loading, and safe tensors format preventing security vulnerabilities by avoiding executable code in model files. Model files contain config files specifying architecture parameters and weights in safe tensor format.

Click to watch →

Attention & Co- Getting started with Transformers

Attention & Co- Getting started with Transformers

2.2K

Hugging Face is an open-source company dedicated to building a community for AI, making state-of-the-art machine learning accessible to everyone. Their ecosystem includes: (1) The Hub - a central repository with 22,000+ models, datasets, and tools, (2) Transformers library - unified API supporting 50+ architectures, (3) Datasets library - optimized for processing large datasets, (4) Accelerate - simplifies multi-GPU training, (5) Spaces - platform for building web applications to demonstrate models. The Transformers library provides a Pipeline API that simplifies common NLP tasks like sentiment analysis, question answering, and translation. Training transformers can be done with just a few lines of code using the Trainer class. The library supports many different architectures using a unified API, allowing practitioners to experiment with different models by simply changing the model name in the code.

Click to watch →

‫آموزش ترنسفورمرز هاگینگ فیس (Transformers Hugging Face): فراخوانی مدل

‫آموزش ترنسفورمرز هاگینگ فیس (Transformers Hugging Face): فراخوانی مدل

361

This comprehensive section covers the foundational knowledge for working with Hugging Face AI models in Python. It introduces Hugging Face as a platform for accessing state-of-the-art AI models and explains the Transformers library for loading them. The video covers installing Transformers (pip install transformers for local, pre-installed in Colab), exploring Hugging Face libraries (Transformers, Datasets, Tokenizers, Sentence Transformers), and understanding the two main model loading methods: Pipeline (simpler, recommended for beginners) and AutoClass (more detailed, more code). The section demonstrates practical implementation using Koala model, showing how to select models by category and parameter count, create pipelines with task='text-generation', and pass input text. It covers different task types supported by Pipeline including audio-classification, depth-estimation, image-text-to-text, and masked-language-modeling. The video emphasizes hands-on practice with exercises for testing with ModernBERT and other models.

Click to watch →

Finetune LLMs to teach them ANYTHING with Huggingface and Pytorch | Step-by-step tutorial

Finetune LLMs to teach them ANYTHING with Huggingface and Pytorch | Step-by-step tutorial

133.4K

The Hugging Face Transformers library provides APIs for downloading and training state-of-the-art pre-trained models. Users specify model IDs to load tokenizers and models. Causal language models predict next words based on previous context, unlike masked models. Tokenization converts text to integer tokens, requiring padding for batch processing. Attention masks filter out padding tokens during computation. Chat templates format prompts with system/user/assistant roles for instruction tuning.

Click to watch →

Fine-Tune vs RAG

RAG vs Fine-Tune

Project Setup

Model Selection

Data Formatting

QLoRA Training

Training Setup

Model Saving

Results Compare

Closing Notes

Fine-Tune vs RAG

0:00

Playing Section

1
Explains when to choose fine-tuning over retrieval-augmented generation.
2
Highlights fine-tuning benefits: better performance and less prompt engineering.
3
Notes drawbacks: requires resources, time, and high-quality data.

Deep Dive

Techniques for deploying and serving fine-tuned LLMs in production environments using frameworks like vLLM, TGI, or TensorRT-LLM.

LLM Inference: A Comparative Guide to Modern Open-Source Runtimes | Aleksandr Shirokov, Wildberries

LLM Inference: A Comparative Guide to Modern Open-Source Runtimes | Aleksandr Shirokov, Wildberries

496

Deploying large language models at scale requires understanding that no single LLM serving framework is optimal for all scenarios; the best approach depends on specific use cases, model sizes, latency targets, and workload patterns, with key optimization techniques including attention backend selection (Flash Attention vs. Flash Attention Free), speculative decoding, key-value caching, and efficient weight loading, while production deployment demands custom benchmarking, proper GPU resource sharing, horizontal autoscaling with cache-aware metrics, and API gateway integration for security and governance.

Click to watch →

From model weights to API endpoint with TensorRT LLM: Philip Kiely and Pankaj Gupta

From model weights to API endpoint with TensorRT LLM: Philip Kiely and Pankaj Gupta

4.9K

TensorRT LLM is Nvidia's high-performance inference framework for large language models, built on TensorRT GPU SDK. It optimizes neural network graphs through pattern discovery and CUDA kernel optimization, with a plugin mechanism for advanced optimizations like Flash Attention. Key features include inflight batching (10-20x throughput improvement), support for multiple GPU architectures (Volta, Turing, Ampere, Hopper), and production-ready deployment. The framework supports approximately 50 foundation models and their fine-tunes, including audio transcription models like Whisper due to shared Transformer architecture. Building optimized models requires extensive production environment information including sequence lengths, batch sizes, and GPU types. Engines are not portable across GPUs. Key performance metrics include tokens per second (30-50 tokens/sec feels fast) and time to first token (sub-200ms feels instant). For Mistral 7B, TensorRT LLM achieved 216 tokens/sec and 180ms time to first token. FP8 quantization provides approximately 40% performance improvement over FP16 with minimal accuracy loss, supported by H100 and L4 GPUs. KV cache quantization is critical because the context phase is compute-bound. Building a TensorRT LLM engine involves configuring GPU, installing dependencies, setting engine parameters, running the build job, and testing results. Cloud GPU platforms like RunPod provide accessible infrastructure for LLM development.

Click to watch →

Compound AI Systems with Philip Kiely - Weaviate Podcast #105!

Compound AI Systems with Philip Kiely - Weaviate Podcast #105!

878

TensorRT-LLM and vLLM are leading frameworks for optimizing LLM inference performance. TensorRT-LLM creates GPU-specific optimized CUDA instructions through engine-building, offering substantial performance gains especially on H100 GPUs, with excellent inflight batching capabilities. vLLM provides simpler developer experience. Both frameworks address making high-performance model serving accessible without deep GPU expertise. Model serving frameworks like Truss simplify packaging Python code with Transformers into Docker containers for deployment on platforms like Base 10, abstracting distributed infrastructure complexity.

Click to watch →

How Fine-tuning Open Source LLMs Solves GenAI Productionization - Piero Molino | Stanford MLSys #94

How Fine-tuning Open Source LLMs Solves GenAI Productionization - Piero Molino | Stanford MLSys #94

3K

This section presents the complete technical infrastructure enabling practical deployment of fine-tuned LLMs at scale. Ludwig is introduced as a declarative machine learning framework that simplifies fine-tuning through YAML-based configurations, abstracting away implementation complexity. The presentation then details Lorax's innovative multi-tier caching architecture: GPU memory stores active adapters, CPU memory caches thousands more, and disk provides on-demand loading. This design enables serving hundreds of fine-tuned models on single GPUs while maintaining near-constant throughput and low latency per request. The section demonstrates how sharing base model weights across all fine-tuned versions dramatically reduces serving costs compared to dedicated deployments per model. Together, these technologies transform fine-tuning from an experimental curiosity into a viable production strategy, enabling organizations to deploy specialized models at scale without prohibitive infrastructure costs.

Click to watch →

I Tested All 4 LLM Deployment Methods So You Don't Have To | Ollama, LLama.cpp, LM studio, vLLM

I Tested All 4 LLM Deployment Methods So You Don't Have To | Ollama, LLama.cpp, LM studio, vLLM

103

vLLM enables production-grade LLM deployment handling multiple concurrent API requests efficiently. Deployment requires creating GPU instances (A100 with 80GB VRAM), installing via pip, and resolving dependency conflicts. Commands specify model IDs, hosts, ports, context sizes, GPU counts, and model parameters. vLLM optimizes for commercial applications serving multiple users simultaneously. Framework selection depends on use case: LM Studio suits casual users with GUI simplicity; llama.cpp offers developer flexibility; vLLM handles production-scale serving. Most individual users should choose between LM Studio for ease or llama.cpp for customization.

Click to watch →

Deep Dive

Implementing model alignment strategies such as Direct Preference Optimization (DPO) or Reinforcement Learning from Human Feedback (RLHF).

Direct Preference Optimization: Forget RLHF (PPO)

Direct Preference Optimization: Forget RLHF (PPO)

16.1K

Direct Preference Optimization (DPO) is a new training methodology developed by Stanford University that bypasses the complex two-step process of Reinforcement Learning from Human Feedback (RLHF) by directly optimizing language models using a single maximum likelihood objective based on human preference data, achieving similar or better performance than traditional RLHF methods while being substantially simpler to implement and train.

Click to watch →

CS 285: Eric Mitchell: Reinforcement Learning from Human Feedback: Algorithms & Applications

CS 285: Eric Mitchell: Reinforcement Learning from Human Feedback: Algorithms & Applications

8.5K

Reinforcement Learning from Human Feedback (RLHF) is a four-step pipeline for training large language models: (1) unsupervised pre-training on internet text, (2) supervised fine-tuning on human demonstrations, (3) fitting a reward model using preference data collected from human comparisons of model responses, and (4) reinforcement learning with the reward model and KL-divergence constraint to optimize the policy. Direct Preference Optimization (DPO) simplifies this process by deriving a closed-form optimal policy directly from preference data, eliminating the need for separate reward model training and RL optimization loops. DPO achieves this by parameterizing the reward function as the log probability ratio between the current policy and a reference model, transforming the RLHF objective into a tractable classification loss that can be optimized directly without rollouts.

Click to watch →

Aligning LLMs with Direct Preference Optimization

Aligning LLMs with Direct Preference Optimization

33.9K

Direct Preference Optimization (DPO) is a simplified alignment technique that trains language models directly on human-preference data without requiring a separate reward model, using a differentiable loss function that maximizes the difference between log probabilities of preferred versus dispreferred responses; this approach achieves comparable or better results than traditional reinforcement learning from human feedback (RLHF) while being more computationally efficient and easier to implement, as demonstrated by its successful application in training Zephyr, a state-of-the-art 7B parameter chat model.

Click to watch →

Reinforcement Learning, RLHF, & DPO Explained

Reinforcement Learning, RLHF, & DPO Explained

15.4K

This video explains three optimization techniques for aligning AI language models with human preferences: (1) RLHF uses reinforcement learning where a reward model scores outputs and an optimizer updates the policy model through trial and error, but faces challenges with instability and expensive human labeling; (2) DPO directly optimizes the language model on preference data without explicit reward modeling, achieving equivalent or better results than RLHF while being simpler, faster, and more robust to temperature changes; (3) KTO applies prospect theory from behavioral economics to maximize utility by considering human risk aversion, using a sigmoid value function and handling noisy/inconsistent data better than other methods, with tunable hyperparameters for balancing risk aversion between desirable and undesirable examples.

Click to watch →

Direct Preference Optimization (DPO) - How to fine-tune LLMs directly without reinforcement learning

Direct Preference Optimization (DPO) - How to fine-tune LLMs directly without reinforcement learning

29.4K

Direct Preference Optimization (DPO) is a method for fine-tuning Large Language Models using human feedback without requiring reinforcement learning. Unlike Reinforcement Learning with Human Feedback (RLHF), which uses two separate neural networks (a policy network and a reward model), DPO trains only the Transformer model directly by embedding the reward function within its loss function. The Bradley-Terry model converts human reward scores into probabilities using the sigmoid function, while KL Divergence prevents the model from changing too drastically from its original behavior. This approach is more efficient than RLHF as it eliminates the need to train a separate reward model.

Click to watch →

Deep Dive

Scaling up fine-tuning to larger models and multi-GPU setups using distributed training frameworks like DeepSpeed or PyTorch FSDP.

Lec 13 | Efficient LLMs: Part 03

Lec 13 | Efficient LLMs: Part 03

434

DeepSpeed provides FSDP (Fully Sharded Data Parallel) for sharding models across GPUs. PyTorch's native FSDP support allows sharding models layer by layer. Distributed samplers split datasets across GPUs using world_size and rank. Gradient accumulation enables processing multiple micro-batches before updating weights. Gradient checkpointing reduces activation memory by recomputing activations during backward pass instead of storing them.

Click to watch →

Fast Finetuning of Gemma-3, Qwen-3 and GPT-OSS on Strix Halo using Unsloth and Multi-Node Setups

Fast Finetuning of Gemma-3, Qwen-3 and GPT-OSS on Strix Halo using Unsloth and Multi-Node Setups

4.1K

Fully Sharded Data Parallel (FSDP) enables training of extremely large language models (such as 27 billion parameter models) that cannot fit on a single GPU. By sharding model weights, gradients, and optimizer states across multiple GPUs, FSDP allows effective utilization of distributed computing resources. Training such large models with FSDP and gradient accumulation can achieve reasonable training times (approximately 50 minutes for 27B parameter models), demonstrating the scalability benefits of distributed training approaches.

Click to watch →

Too Big to Train: Large model training in PyTorch with Fully Sharded Data Parallel

Too Big to Train: Large model training in PyTorch with Fully Sharded Data Parallel

1.8K

Fully Sharded Data Parallel (FSDP) is a PyTorch library that enables training of very large models by sharding model parameters, optimizer states, and gradients across multiple GPUs, trading increased communication overhead for reduced memory footprint; unlike standard Distributed Data Parallel (DDP) which replicates the entire model on each GPU, FSDP allows each GPU to store only a subset of the model, making it possible to train models that exceed the memory capacity of any single GPU, though careful configuration of sharding strategies and auto-wrap policies is essential to balance memory savings against communication costs.

Click to watch →

Как мы ускоряем обучение нейросетей с YaFSDP / Антон Фролов, Яндекс

Как мы ускоряем обучение нейросетей с YaFSDP / Антон Фролов, Яндекс

769

Large language models like GPT-4 require millions of dollars to train due to scaling laws: longer training, more parameters, and more data produce smarter models. State-of-the-art models contain over 100 billion parameters and require trillions of tokens. Single GPU training suffices for education but cannot handle modern large models. Distributed training enables training across multiple GPUs. Data parallelism processes micro-batches with AllReduce averaging gradients, but each GPU stores a complete model copy. Model parallelism with weight sharding divides weights and optimizer states across GPUs, reducing memory usage proportionally. Three major frameworks evolved: DeepSpeed ZeRO (2020) implements staged sharding with asynchronous communication; FSDP (2022) groups parameters for better communication, achieving 30% speedup; YaFSDP solves memory locality and 'pineapple effect' through systematic optimization. YaFSDP achieves 2-20% speedups across Llama models on 8-256 GPUs. Open source release with Hugging Face integration enables broader adoption. Economic analysis shows 20% speedup on 6.4 million GPU hours saves $3.2 million, demonstrating substantial value for both large-scale training and iterative experimentation.

Click to watch →

Scaling AI Model Training and Inferencing Efficiently with PyTorch

Scaling AI Model Training and Inferencing Efficiently with PyTorch

8.2K

FSDP addresses the limitation of DDP when models exceed single-GPU capacity by sharding the model into units distributed across GPUs. During forward pass, units are gathered via AllGather, computed, and then released from memory. The backward pass follows the reverse pattern. Unlike DDP's single synchronization point, FSDP provides multiple opportunities for overlapping computation and communication, enabling efficient training of very large models that cannot fit on a single GPU.

Click to watch →

Deep Dive

Designing hybrid architectures that combine fine-tuning with Retrieval-Augmented Generation (RAG) to reduce hallucinations and leverage external knowledge.

Satelliser des IA : on vous explique le nouveau pari fou d'Elon Musk

Satelliser des IA : on vous explique le nouveau pari fou d'Elon Musk

42.6K

Retrieval-Augmented Generation (RAG) is a technique that reduces AI hallucinations by providing the model with relevant source documents during generation. By grounding responses in actual source material, RAG reduces the model's tendency to invent information. This approach has become standard practice in production AI systems, with models like Grok and Claude implementing multiple layers of RAG to minimize hallucinations.

Click to watch →

Agentic AI – Đỉnh cao mới của AI sau GenAI | Trần Quốc Huy - WecommitAI

Agentic AI – Đỉnh cao mới của AI sau GenAI | Trần Quốc Huy - WecommitAI

23.9K

RAG (Retrieval-Augmented Generation) addresses GenAI's hallucination problem by connecting AI to external knowledge sources. Instead of generating answers solely from training data, the AI first retrieves relevant information from provided databases or documents, then generates responses based on that retrieved information. This approach allows businesses to use their proprietary data, ensuring accurate responses about company policies, products, and internal information while reducing hallucinations. Google and Facebook independently proposed similar solutions in 2020, with Facebook's approach becoming more widely adopted in practice.

Click to watch →

A Survey of Techniques for Maximizing LLM Performance

A Survey of Techniques for Maximizing LLM Performance

220K

For certain use cases, combining fine-tuning and RAG provides the best results. Fine-tune the model to understand complex instructions, eliminating the need for complex prompt engineering at inference time. This frees up context window space for RAG-injected relevant knowledge. The combined approach minimizes prompt tokens while maximizing available context for dynamic knowledge injection, creating an efficient pipeline where the model's understanding is baked-in and additional context can be added as needed.

Click to watch →

ChatGPT erklärt die Welt? • Generative KI: Werkzeug und Weltveränderer | Martin Schiele

ChatGPT erklärt die Welt? • Generative KI: Werkzeug und Weltveränderer | Martin Schiele

42K

To make language models useful for specific domains, organizations can either fine-tune the model on their own data or use Retrieval Augmented Generation (RAG). Fine-tuning involves training the model on domain-specific data (like company documents, chat histories, or technical manuals) to adapt its behavior. However, this approach requires significant data preparation, hardware resources, and creates challenges when new data needs to be incorporated. Fine-tuned models may also exhibit more hallucinations because they attempt to recall information from memory rather than referencing actual documents. RAG offers an alternative by keeping the base model unchanged while providing it with relevant documents at query time. This approach is more dynamic, requires less computational resources, and produces fewer hallucinations because the model can reference actual documents rather than relying solely on its compressed training memory.

Click to watch →

Day 2 Livestream with Paige Bailey – 5-Day Gen AI Intensive Course | Kaggle

Day 2 Livestream with Paige Bailey – 5-Day Gen AI Intensive Course | Kaggle

64.8K

RAG addresses LLM limitations by combining retrieval with generation through three stages: (1) embedding and indexing documents in a database, (2) retrieving semantically similar documents for queries, and (3) providing retrieved documents as context for LLM answer generation. This approach improves performance and reduces hallucinations by providing relevant, up-to-date information. The implementation uses ChromaDB for vector storage and Gemini API for embedding and generation.

Click to watch →

Fine-Tune vs RAG

RAG vs Fine-Tune

Project Setup

Model Selection

Data Formatting

QLoRA Training

Training Setup

Model Saving

Results Compare

Closing Notes

Fine-Tune vs RAG

0:00

Playing Section

1
Explains when to choose fine-tuning over retrieval-augmented generation.
2
Highlights fine-tuning benefits: better performance and less prompt engineering.
3
Notes drawbacks: requires resources, time, and high-quality data.

135.7K views2K18:28@venelin_valkovOriginal Release: 2023-09-04

Fine-tuning the Llama 2 7B base model using the QLoRA technique enables training a custom text summarization model on conversation data (such as Twitter support agent interactions) with minimal computational resources, achieving significantly better performance than the base model by adapting the model to specific domain patterns through low-rank adaptation.

[00:00:00]how can you improved the performance of GPT light models for your specific use case one approach is to fine tune the watch language model such as Walmart 2.0 using your own data set so when should you fine-tune a large language model such as Walmart 2.0 well the answer for that is that when you have to do it so when prompting doesn't work for your use case and when you know how to do and what to do when you're using a watch language models with your own data you basically have two approaches that you can integrate the watch language model with your own data sets on the left we have the fine tuning approach and on the right we have the retrieval augmented generation which is probably the more popular approach since it requires just the prompt and then you have to basically input some parts of your text right within the prompt so why you may use a rack or retrieval augmented generation the simple way to answer this is that you have an easier way to use multiple knowledge bases or just single one so for example if you have some financial data or let's say wire documents Etc multiple PDFs you can use those to integrate some parts of the text right within the prompt and analyze that using the watch language model we have a single model that you can essentially ask it a multiple tasks and this will work with just this one model and then some of the drawbacks of this approach is that it really doesn't have much more knowledge or any other knowledge on your data outside of what you're putting into the prompt and then you are pretty much have to experiment with multiple prompts and evaluate their Performance Based on how good your prompts can be on or how good your prompt engineering techniques can be and getting a consistent type of output for example markdown or Json or some different approach to the formatting output can be a bit of a hard task to do as well so fine tuning is actually solving those two problems pretty much in the general case and but it is of course adding some more complexities to the overall performance of the model so benefits of the fine-tuning a large language model is that probably the model if fine tuned correctly is going to have much better performance on your own tasks or use cases compared to the general order based large language model then you have to do a lot more or a lot less prompt engineering so you basically have to put in some prompts that are that can be much shorter compared to what you have to do within the general watch language model and then the result of that is that you will probably have much more tokens that are available based on the limits of the model and those tokens you can use in order to provide even more data of course but the drawbox drawbacks of the fine tuning a watch language model is that it can be hard to do this requires a lot of resources a lot of time to experiment and then you of course have to have a lot of high quality data in order to get some good results on the fine tuning and if you have or if you plan to use this with external knowledges such as when you're doing retrieval augmented generation this might not work as well as the general large language model since it is fine on your specific cases in this video I'm going to show you an example on how to fine tune a watch language model Walmart 2.0 in our case in order to summarize conversations on Twitter between customer support agents and users and the task will be to summarize conversations into something that is easier to read and short of course so we are going to use a custom data set in order to do that if you want to follow along there is a complete text tutorial called fine-tuning Walmart 2.0 on a custom date set that is available on ML expert dot IO and this one is available for ML expert Pro subscribers only so here you can also find a link to the Jupiter notebook or the Google clock notebook along with a complete tutorial along with the code and examples and you can pretty much follow along everything that we are doing here so if you want to support my work please consider subscribing to ml expert Pro thanks of course the data set that you're going to use is going to be specific to your own use case and in our example I am taking this data set from Salesforce dialogue studio so this is pretty much a collection of unified dialogues data set and instruction aware models for conversation AI provided Again by Salesforce and from here I'm using this Twitter Sim data set which contains these splits so those already for us to use and you can see that we have some summaries right here and we have the conversations themselves so I'm going to continue with using or I'm going to show you how you can process this data in order to provide it into a format that is available for training alarm two point model the next logical step is to choose a large language model that you can fine tune on your own data and for that you'll probably need something that is open and you can use it in a commercial setting so here we are going to use pretty much the state of the art open model provided by meta AI called Obama 2.0 and this module has a lot of advantages it has different model sizes and it is having a context length of over 4000 context are tokens so yeah this is pretty much state of the art model right now and this is visible on the open Arrow leaderboard where pretty much every top spot right here is the fine tune version of this Lama 2.0 model I have a Google Cloud notebook that is already running and here you can see that we are using a Tesla T4 GPU with about 16 gigabytes of vram I'm not using a high Ram option so this should be available for Google Co-op free users as well and I'm installing pretty much the standard Library storage Transformers data sets but then we're going to use a water technique and or QR then bits and bytes and then the TRL or Transformers reinforcement warning library for the trainer I'm going to show you that in a bit we have the Imports and most of this is again pretty standard for loading the model and then we're going to use a bits and bytes config for the quantization that we going to do with the Q order technique training arguments and then the trainer Itself by the TRL Library I'm going to use the Walmart 2.0 7 billion base model so not the instruction or chat tuned model so this is going to do just the summarizations without the instruction training and here we are loading the data set that I've shown you in a bit a while back so I'm going to do some pre-prosync in order to extract conversations from this and I'm going to show you the resulting conversation that I'm going to Output from a single example you can see that the conversation is formatted into a user agent user agent conversation we have also the summary and then we have the instruction that we are going to pass in as a prompt to our model note that we're using an Outback style formatting of the template or the prompt we have the instruction below is the conversation between a human and an AI agent write a summary of the conversation then we have the input with the conversation itself and then this is the response or the summary so if you want to use your own personal data set you might have to format it into a similar format of course the alpaca style is not required but I find it pretty easy to use and read so I prefer to format my fine tuning examples in similar format but note that the base model doesn't actually use any prompt formatting so you might use whatever you like so in order to find out to transform this data set into that I'm creating this function called generate text and the most important part here is that I'm getting the summaries I'm getting the first summary of the of the summaries and then I'm creating the conversation text with just which just iterates over the output of the conversation and I'm using this clean text function in order to remove URL mentions in Twitter multiple spaces and then some strange characters that I found and I'm pretty much using the user and the agent in order to provide the conversation text itself and finally I'm using this generate training prompt which is using the default system prompt that I've shown you uh it is be always a compensation between a human and an AI agent right in the summary of the conversation of course you can try something different as well and I'm just formatting the system prompt the conversation and the summary into the alpaca format that we are using and this is the result that we are getting so I'm pretty much going over the whole data set shuffling it then calling this function and removing all of the original columns since we don't actually need those and this is the data set for the training and validation and test split thus far we have just the conversation the summary and the text for each point then I'm going in Into The Notebook on the hugging face and I'm then downloading the model which again is awama 2.0 with the 7 billion parameters and I'm loading this into a normalized Fallout 4 so this is the quantization that we are going to use and then I'm getting the tokenizer for the warmer this should take some time and then uh what is pretty nice about the recent versions of the Transformers library is that you have this quantization config right within the models so this is Now supported and it works with bits and bytes uh also it works with auto gptq if you're uh going to use that you might give it a try this should support it as well and then I'm using a war config which is very similar to what I did previously we have the ranking of the Matrix equal to 16 and I have a scaling factor of 32 and some Dropout you can play around with these settings and those might work better on your own data sets if you change those but I'm using this causal watch language modeling for the task that we're going to do so this is going to just predict the next token and I did run this for about let me see yeah this all run for about 12 minutes on a Visto V100 GPU actually on the T4 is going to probably take two or three times more but still very feasible and this is uh the response or the yeah the evaluation was that I got let me just have a look at the scalers right here yeah so you can see that the evolution was was running very nicely and probably if you are running this for more than two epochs this is what I did you might get even better validation loss the training was was also trending downwards so I would say that and this training was pretty okay for just the two epochs we were able to converge to some nice validation was so this was good and in order to train this I was been using these training arguments uh you can note those parameters right within the tutorial or the full text tutorial that is available for on ML expert one of the more important things here is that I'm using a cosine ordering schedule so you can see that actually the yeah actually the warning crates during the training yeah and you can see that we were using this specific warning create scheduler and this seems to be working quite well at least for this example so I would uh suggest that you use similar worrying worrying rate schedule as well and the other important thing is here that we are using the bits and bytes compatible atom Optimizer with the weight Decay fix so this is a pretty nice and it appears to be working quite well at least for this training so another thing is that I'm passing in the path config or the config for the water or QR right within the SS sft trainer so this is the trainer that accepts the training arguments we are giving it a maximum sequence length of the that is equal to the Walmart 2.0 and then the field that we are going to use for the training in our case this is the text field so this contains the prompt in the correct format of course and then this is a very nice table that we got during the training and you can see that the validation was was converging nicely of course the training was was uh getting down a lot faster yeah but either the validation or the test set are pretty uh small so it's hard to tell whether or not the results are quite good at least looking at those but at least we're looking at some type of convergence and minimization so after drink is complete I am just saving the model and this will just save the keyword adapter so if you want to merge this you might use this code in order to find the water or the adapter and apply it to the whole model itself so you can have a single model without the need to apply apply the keyword adapter to it either way for the inference and uh comparison between the fine tune and the base model I'm just generating this prompt with the instruction the conversation and then asking for the summary with the correct alpaca format and I took a lot of examples or just five of the examples within the test data set so those are examples that the model haven't seen and I just formatted them into the way that we've formatted the training set so I've created the model and then rolled this small function that pretty much summarizes some text by passing in the tokenizer and then tokenizing the text and then I'm pretty much cutting out the generated outputs after the input length on the tokenizer using a very small temperature size and maximum new tokens equal to 256.

[00:15:46]so in order to have a look at a single example of this base model I took this conversation and this summary and we took about 24 seconds in order to summarize this conversation and the response of the base model is pretty much uh junk yeah so you can see that we are getting pretty much the same input that we the same output as the input of the conversation so not very good the second example is pretty much the same thing the conversation between change the phone number of an account yeah I think that this is actually Within with the fine-tuned version of the model not sure about that and here we have another response that is uh pretty much the input itself so not very good let's have a look at the fine-tuned version of the outputs so for example with this conversation uh we got this response and if you just take this part we have a very long summary but still a summary that contains all the information of the conversation so I would say that the here the fine-tuned model performed much better compared to the base model but not great since this summary is very long but on the second example I just took all of the output up to the First new line text and here the summary is much better customer is complaining that his account is linked to an old number and now he is asked to verify his account so yeah the summary of this conversation is very good at least this is with the fine-tuned version and then for the third example customer is complaining about the new updates on iOS 11 and we have the conversation right here you can read this for yourself so the predicted summary is customer is complaining that the new update iOS 11 6 agent asked to DM and they will work from there so again pretty nice summary so the fine tune version of this model is performing very well if you found this tutorial helpful please let me down in the comments below also if you have any further questions use the comments and please like share and subscribe to this video also please join the ml expert Pro subscribers to get a full text tutorial on the Google Co-op notebook thanks

Up Next

Agentic LLMs & RAG: Tools, ReAct, and Retrieval (Lecture 7) - Stanford CME295

@stanfordonline

66K views•2025-11-18

Building Real-Time ML Pipelines with Feature Stores and MLOps Frameworks

@ODSCAI

5.1K views•2022-02-20

Bypassing Tor Censorship: Bridges and Pluggable Transport Guide

@Coding_ForEveryone

397 views•2024-06-11

Neural Networks Explained: Math, Layers, and Learning Fundamentals

@3blue1brown

21.9M views•2017-10-05

Related Study Plans & Knowledge Roadmaps

Structured learning paths in Artificial Intelligence

View All Artificial Intelligence Topics

artificial-intelligence

Building Recommenders: Two-Tower & Vector DBs

01Foundations of Python & Machine Learning Basics

02Introduction to Recommender Systems

03Deep Learning & Representation Learning

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

Latent Diffusion: PyTorch, UNet & CLIP

01Deep Learning & PyTorch Foundations

02Convolutional Networks & Autoencoders

03The Math and Mechanics of Diffusion Models

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

NeRFs: 3D Reconstruction & Instant-NGP

01Math & Python Foundations for 3D Vision

02Photogrammetry & Camera Pose Estimation

03Neural Radiance Fields (NeRF) Core Concepts

+ 2 more learning modules

Explore Roadmap

artificial-intelligence

TFT Forecasting: Energy Demand & Deep Learning

01Programming & Data Science Foundations

02Introduction to Machine Learning & Time Series

03Deep Learning & Sequential Models

+ 3 more learning modules

Explore Roadmap