Academic Databases
Explore Our Semantic Databases
STEM & Tech
Math, Physics, CS & Engineering...
Economics
Macro, Micro, Econometrics...
Business
Management, Marketing, Finance...
Health & Life Sciences
Biology, Medicine, Neuroscience...
Humanities & Social Sciences
Psychology, Law, Philosophy, History...
General Studies
General & interdisciplinary studies...
Explore Topics
Features

AI Quantization Explained: Optimize Local LLM Performance

Added: 2026-08-07

Deep Dive

Understanding of Large Language Model (LLM) architecture and how model weights (parameters) represent learned information.

What Are Large Language Models, Really? | Inside Modern AI Systems: An Engineer’s Guide | Part-1

What Are Large Language Models, Really? | Inside Modern AI Systems: An Engineer’s Guide | Part-1

825

Large Language Models (LLMs) are fundamentally collections of numerical weights (hundreds of thousands of GBs) combined with a fixed transformer architecture, where intelligence emerges from how these weights are processed through the transformer forward pass algorithm rather than from the weights themselves; the inference engine loads these static weights into GPU memory and executes the transformer algorithm on input tokens to generate outputs, with the model weights remaining unchanged during inference and only modified during the training phase.

Click to watch →

Лучший доклад по GEO! Видимость брендов в LLM (ИИ) — Алексей Чекушин

Лучший доклад по GEO! Видимость брендов в LLM (ИИ) — Алексей Чекушин

518

LLMs (Large Language Models) are built on the Transformer architecture, introduced in 2017. They consist of a large number of parameters (weights), which are floating-point numbers. For example, GPT-4 has approximately 1.7 trillion parameters, while Grok-1 has 314 billion parameters. These parameters represent the entire knowledge and intelligence of the model, as there is nothing else in the model.

Click to watch →

CURSO de DESARROLLO con IA 2026 (con Certificado) - Brais Moure

CURSO de DESARROLLO con IA 2026 (con Certificado) - Brais Moure

4.9K

This section provides a deep dive into Large Language Models (LLMs), explaining that they are AI models designed to understand and generate text or code. The section covers how LLMs function through two main processes: training (learning from vast amounts of data) and prediction (predicting the next word or token in a sequence). The number of parameters represents neural connections that store learned knowledge, with more parameters generally meaning better ability to represent complex knowledge. The section also explains tokens as basic units for processing text, directly related to costs, and the context window as the AI's short-term memory that stores conversation information.

Click to watch →

Instruction fine-tuning: Loading pre-trained LLM weights

Instruction fine-tuning: Loading pre-trained LLM weights

7.9K

Large language models consist of multiple components with trainable weights including multi-head attention mechanisms (query, key, value matrices), feed-forward neural networks, layer normalization layers with scale and shift parameters, token embedding layers, and positional embedding layers. Together, these components contain over 100 million parameters in larger models like GPT-2. Pre-trained models like GPT-2 are trained on massive datasets and have their weights publicly released by organizations like OpenAI in various sizes (124M, 355M, 774M+ parameters). Reusing these pre-trained weights allows models to start from a knowledgeable state rather than random initialization, dramatically reducing training time and computational resources required for downstream tasks.

Click to watch →

Используем C для исследования LLM-мира • Шаг первый — токенизация • Live coding

Используем C для исследования LLM-мира • Шаг первый — токенизация • Live coding

1.2K

LLMs consist of parameters (weights) that define how the model processes information, and an architecture that defines how these parameters are organized and used. The parameters are the learned values that the model uses to make predictions, while the architecture determines how these parameters interact. Together, they define the model's capabilities and behavior. Understanding this distinction is important for anyone working with LLMs.

Click to watch →

Deep Dive

Basic computer data representation concepts, particularly the difference between floating-point precision formats (FP32, FP16) and integer formats (INT8).

Lecture 15 | Efficient Methods and Hardware for Deep Learning

Lecture 15 | Efficient Methods and Hardware for Deep Learning

188.9K

Floating point numbers are discrete representations, not continuous real numbers. FP32 uses 32 bits with three components: sign bit, exponent bit, and mantissa. The value is calculated as (-1)^s × 1.M × 2^exponent. FP16 uses 16 bits instead of 32, reducing both energy consumption and chip area by approximately four times. Int8 represents fixed-point numbers using integer bits followed by a radix point and fractional bits, commonly used in TPUs for efficient computation.

Click to watch →

Mixed Precision Training | Explanation and PyTorch Implementation from Scratch

Mixed Precision Training | Explanation and PyTorch Implementation from Scratch

997

FP16 uses 16 bits with 5 exponent bits and 10 mantissa bits, severely limiting both range and precision. BF16 maintains FP32's 8-bit exponent but reduces mantissa to 7 bits, preserving wide range while sacrificing precision. Pi becomes 3.140625 in both lower formats versus full decimal in FP32. Epsilon quantifies precision—the smallest number adding to 1 produces a larger result. FP32 offers maximum range with finest tick marks; BF16 maintains similar range with coarser precision; FP16 has limited range with slightly finer but still coarse ticks. This trade-off between range and precision drives format selection.

Click to watch →

Is Tesla Building the World’s Best Supercomputer? + TSLA Stock Hits All-Time Highs

Is Tesla Building the World’s Best Supercomputer? + TSLA Stock Hits All-Time Highs

65.2K

FP32 (floating point 32-bit) and FP16 (floating point 16-bit) represent different precision levels in computer calculations. FP32 uses 32 bits of memory to store numbers, providing more significant digits and greater precision. FP16 uses only 16 bits, storing approximately half the number of significant digits, resulting in less precise calculations. While FP16 enables faster calculations due to reduced memory usage, FP32 provides better accuracy for applications requiring high precision.

Click to watch →

Part 1-Road To Learn Finetuning LLM With Custom Data-Quantization,LoRA,QLoRA Indepth Intuition

Part 1-Road To Learn Finetuning LLM With Custom Data-Quantization,LoRA,QLoRA Indepth Intuition

153.4K

FP32 (Floating Point 32-bit) represents full precision or single precision floating point numbers, commonly used for storing neural network weights during training. FP16 (Floating Point 16-bit) represents half precision, which uses fewer bits to store the same type of numerical values. The conversion between these formats is a fundamental aspect of quantization, where FP32 values are compressed into FP16 to reduce memory usage while attempting to preserve model accuracy.

Click to watch →

EfficientML.ai Lecture 5 - Quantization (Part I) (MIT 6.5940, Fall 2023, Zoom recording)

EfficientML.ai Lecture 5 - Quantization (Part I) (MIT 6.5940, Fall 2023, Zoom recording)

7.2K

FP32 (32-bit floating point) consists of three parts: 1 sign bit, 8 exponent bits, and 23 fraction (mantissa) bits. The number is represented as (-1)^sign × (1 + fraction) × 2^(exponent - bias), where bias = 127 for 8-bit exponents. This format provides a much larger dynamic range than integer representation while maintaining precision near zero.

Click to watch →

Deep Dive

The fundamentals of model inference and the hardware constraints (VRAM, system RAM, bandwidth) associated with running AI models locally.

STOP. You're Installing Local AI Right Now (Llama.cpp)

STOP. You're Installing Local AI Right Now (Llama.cpp)

5.8K

Local AI does not require supercomputers or expensive hardware. A decent gaming computer can run local AI models effectively. Small models can rival cloud-based models like Claude 4.5 in intelligence. With 60GB of VRAM and 24GB of RAM, medium-sized models can be run locally. Local AI offers advantages including lower cost (only electricity costs), privacy, and the satisfaction of owning your AI infrastructure. Inference engines like llama.cpp, LM Studio, Lemonade, and KoboldCPP provide different user experiences. Model selection is important: Qwen 3.5/3.6 excels at coding and reasoning, while Gemma 4 is better for writing, productivity, and general chatting. Model parameters (measured in 'B' for billion) indicate model size and intelligence. MoE (Mixture of Experts) models like 35B are faster and more accessible because they only use 3B parameters per calculation, splitting between VRAM and system RAM. Dense models like 27B require all parameters to fit in VRAM for optimal performance. Memory lobotomy reduces VRAM usage by quantizing KV cache (the memory used for context). KV cache size depends on context length - for example, a 32K context with a 27B model takes about 6-7GB of VRAM. Quantization levels: F16 is default, Q8 reduces KV cache by half, Q5 is a good balance, Q4 is the lowest recommended. KV cache quantization is controversial and can hurt model performance, especially for MoE models and coding tasks. Gemma models are particularly sensitive to quantization.

Click to watch →

ULTIMATE Local Ai FAQ

ULTIMATE Local Ai FAQ

151.8K

For local AI inference, VRAM capacity is the primary performance metric, with the RTX 3090 (24GB VRAM, $850) offering the best value; system bandwidth (measured in GB/s) directly determines inference speed, with CPU-based systems using DDR4 RAM (70-200 GB/s) being significantly cheaper than GPU VRAM ($10+/GB), while multi-GPU setups pool VRAM for larger models but don't proportionally increase inference speed.

Click to watch →

The Only 2 Numbers That Decide Which Local AI Your PC Can ACTUALLY Run

The Only 2 Numbers That Decide Which Local AI Your PC Can ACTUALLY Run

1.1K

Running local AI models on your PC depends on two key hardware metrics: VRAM (video memory on your graphics card) and RAM (system memory). VRAM is critical because AI models prefer fast GPU memory for optimal performance. To find your VRAM, open Task Manager, go to Performance > GPU, and check 'dedicated GPU memory.' If no GPU is listed, your VRAM is effectively zero, and models will run on slower system RAM. Model size is measured in billions of parameters (B), with approximately 0.5 GB of memory needed per billion parameters at normal quality settings. For best performance, models should fit entirely within VRAM; otherwise, they spill onto slower system memory, causing significant slowdowns. Tools like LM Studio and online compatibility checkers help determine which models will run well on your specific hardware before downloading.

Click to watch →

Want to Run AI Agents Locally? Here is The Bare Minimum Setup/Build

Want to Run AI Agents Locally? Here is The Bare Minimum Setup/Build

158K

When building a PC for local AI agents, VRAM (video memory) is the single most critical specification to prioritize over raw processing power, as it determines how much of the AI model can fit in the graphics card's memory; models like 7B, 14B, 32B, and 70B require approximately 5GB, 10GB, 20GB, and 40GB of VRAM respectively, and the system's overall performance depends on having sufficient VRAM to load the entire model without swapping to slower system memory, which causes dramatic slowdowns from smooth operation to unusable speeds.

Click to watch →

La tua SCHEDA VIDEO NON va BENE! Ecco perchè...

La tua SCHEDA VIDEO NON va BENE! Ecco perchè...

25.6K

VRAM functions as specialized storage for graphics processing, analogous to system RAM. The video uses a warehouse analogy: larger VRAM capacity means more data can be stored simultaneously. The memory bus width determines data transfer capacity—wider buses (like RX 580's 256-bit) allow more parallel data transfer, while narrower buses (like RX 5500 XT's 128-bit) require faster individual transfers. Bandwidth represents overall data transfer speed. The RX 5500 XT benefits from newer 7nm architecture and GDDR6 memory, compensating for its narrower bus.

Click to watch →

Deep Dive

A basic grasp of the trade-off between model size, computational speed, and overall output accuracy.

Как заставить нейросеть работать на своем компьютере. Без видеокарты, интернета и бесплатно.

Как заставить нейросеть работать на своем компьютере. Без видеокарты, интернета и бесплатно.

112.3K

There is a fundamental trade-off between model size and inference speed. Larger models with more parameters (13B) provide better quality but initialize slower and generate responses more slowly. Smaller models (7B or 8B) work approximately twice as fast. Users must balance between quality and speed based on their needs - if time is not critical, larger models can be used for better results.

Click to watch →

Network/System Aspects or Operating AI Services[대전대학교 홍용근]

Network/System Aspects or Operating AI Services[대전대학교 홍용근]

175

Deep learning models exhibit fundamental trade-offs between size, accuracy, and inference speed. Larger models like Inception v4 achieve higher accuracy (over 80%) but require approximately 175MB of memory and take about 3 seconds for inference. Smaller optimized versions like MobileNet with quantization achieve around 70-71% accuracy while reducing model size to approximately 1-2MB and inference time to about 200ms. These trade-offs mean practitioners must select appropriate models based on specific application requirements and hardware capabilities.

Click to watch →

Small Data vs Big Data!!!! Why is focusing on Small Data important?

Small Data vs Big Data!!!! Why is focusing on Small Data important?

15.9K

There is often a trade-off between model size, cost, and accuracy in AI/ML applications. Organizations sometimes find that deploying smaller models locally achieves 90% of the accuracy of large cloud-based models at significantly reduced costs (potentially one-tenth the cost). The question becomes whether the remaining 10% accuracy improvement from larger models justifies the substantially higher costs for every query.

Click to watch →

Suchanfragen besser verstehen: Neural Search mit Sentence Transformers | MAIN Session 2.0 – OTTO

Suchanfragen besser verstehen: Neural Search mit Sentence Transformers | MAIN Session 2.0 – OTTO

530

There's a trade-off between model size and inference speed. Larger models (1.1GB) are more accurate but slower, while smaller models are faster but less accurate. The choice depends on the specific requirements of the application.

Click to watch →

QWEN 3.6 🚀 Esta IA local SUPERA a GPT-4o y o3-mini en CÓDIGO

QWEN 3.6 🚀 Esta IA local SUPERA a GPT-4o y o3-mini en CÓDIGO

140

Increasing model size improves accuracy but reduces speed—larger models may make fewer errors but run at only 5% of smaller model speeds. For practical applications, smaller models that run faster may be more valuable than larger, more accurate models. Speculative decoding can further improve speeds to 35+ tokens per second even on older GPUs. The optimal model choice depends on specific use cases: agentic tasks benefit from larger models, while simple development tasks work well with smaller ones. Users should balance accuracy gains against speed losses based on their actual needs.

Click to watch →

Quantization Basics

How It Works

Context Memory

Memory Savings

Model Selection

Action Plan

Quantization Basics

0:00

Playing Section

1
Explains need for precision in AI models and RAM cost.

Deep Dive

Exploring advanced quantization frameworks and file formats, such as GGUF, GPTQ, AWQ, and EXL2, to understand their specific use-cases.

How to Quantize an LLM with GGUF or AWQ

How to Quantize an LLM with GGUF or AWQ

13.6K

Model quantization reduces numerical precision from 32/16-bit to 4-bit representations, enabling LLMs to run on smaller devices and GPUs. Key benefits include reduced VRAM requirements (e.g., Llama 70B from 2-3 A100 GPUs to single A6000 or Mac M1), lower costs, and broader accessibility. Four major approaches differ significantly: (1) Data dependence—AWQ/GPTQ require training data for adaptive quantization, while GGUF/BitsAndBytes are data-independent; (2) On-the-fly support—BitsAndBytes enables dynamic quantization during inference, while data-dependent methods require pre-processing; (3) Speed—GPTQ with act_order is slower than AWQ; (4) Fine-tuning support—BitsAndBytes/GPTQ/GGUF support LoRA, but AWQ lacks this capability; (5) Model sharing—GGUF/AWQ save directly to Hugging Face, while BitsAndBytes requires merging first. Selection depends on hardware (GGUF for laptops, AWQ for GPUs) and use case (fine-tuning vs. inference).

Click to watch →

All You Need To Know About Running LLMs Locally

All You Need To Know About Running LLMs Locally

307K

Running large language models (LLMs) locally involves selecting appropriate software interfaces (such as Text Generation Web UI, Silly Tavern, LM Studio, or Axolotl), choosing suitable model formats (like GGUF, EXL2, AWQ, or GPTQ) that optimize memory usage through quantization, managing VRAM constraints using techniques like CPU offloading, and optionally fine-tuning models with methods like LoRA for specialized applications; the key principle is that model size (measured in billions of parameters) correlates with VRAM requirements, and various optimization techniques enable running larger models on consumer-grade GPUs.

Click to watch →

Reverse-engineering GGUF | Post-Training Quantization

Reverse-engineering GGUF | Post-Training Quantization

51.8K

GGUF (Quantized GGML Unified Format) is a binary file format for storing quantized LLM weights, built on top of GGML (a lean C/C++ tensor library) and llama.cpp (an LLM inference engine). The quantization ecosystem has evolved through three generations: Legacy quants (Type 0 symmetric and Type 1 asymmetric linear quantization using block-based scales and zero points), K-quants (introducing superblocks and double quantization to reduce overhead by quantizing the quantization constants themselves), and I-quants (using vector quantization with codebooks to compress multiple weights into binary codes). The importance matrix, which can be applied to any quantization method, estimates weight importance based on model behavior on calibration data and optimizes dequantization constants to preserve precision for important weights. Mixed precision variants (_S, _M, _L, _XL) allow different bit widths across the model for optimal quality-speed trade-offs.

Click to watch →

Which Quantization Method is Right for You? (GPTQ vs. GGUF vs. AWQ)

Which Quantization Method is Right for You? (GPTQ vs. GGUF vs. AWQ)

37.7K

This tutorial explains three common quantization methods for running large language models on limited hardware: GPTQ (post-training quantization optimized for GPU inference), GGUF (formerly GGML, focused on CPU execution with optional GPU layers), and AWQ (activation-aware weight quantization that prioritizes important weights). The choice depends on hardware constraints—use native loading if GPU capacity permits, GPTQ for moderate GPUs, GGUF for very small GPUs, and AWQ for newer applications where available.

Click to watch →

LLM Quantization Explained: GPTQ, AWQ, QLoRA, GGUF and More

LLM Quantization Explained: GPTQ, AWQ, QLoRA, GGUF and More

260

LLM quantization reduces model memory footprint by converting high-precision weights (FP32) to lower-bit representations (INT8, INT4, etc.), trading precision for computational efficiency; the fundamental trade-off involves balancing memory savings against accuracy degradation, with advanced methods like GPTQ (Hessian-based error compensation), AWQ (activation-aware scaling), and QLoRA (quantized low-rank adaptation) achieving significant compression while maintaining model performance through sophisticated mathematical techniques that compensate for quantization errors.

Click to watch →

Deep Dive

Measuring and evaluating model degradation post-quantization using evaluation metrics like perplexity and standardized LLM benchmarks.

How Do We Get MASSIVE Model To Run On Device? Quantization Explained.

How Do We Get MASSIVE Model To Run On Device? Quantization Explained.

1.2K

This section explains perplexity as a model evaluation metric. Perplexity measures model confidence, not accuracy—a model with low perplexity is highly confident but could still be wrong (a 'confident liar'). High perplexity means the model is uncertain about predictions. Quantization is typically measured against perplexity rather than accuracy because measuring accuracy is more expensive. Small models compressed to one or two bits become essentially random word generators because their perplexity becomes too high. The video emphasizes that perplexity numbers are what users typically see for quantized models, not accuracy benchmarks.

Click to watch →

Does LLM Size Matter? How Many Billions of Parameters do you REALLY Need?

Does LLM Size Matter? How Many Billions of Parameters do you REALLY Need?

44.2K

Reducing precision through quantization does reduce the quality of the model. Data scientists quantify this loss of precision using a metric called perplexity, which measures how confused the model is when making predictions. Lower perplexity indicates the model is better at predicting the next token in a sequence. However, research shows that even with slight increases in perplexity from quantization, the model remains functionally intact and still performs useful tasks.

Click to watch →

Локальные нейросети на MacBook с Киром Белевичем: зачем, как выбрать, MLX, квантование, Dense vs MoE

Локальные нейросети на MacBook с Киром Белевичем: зачем, как выбрать, MLX, квантование, Dense vs MoE

1.7K

Simple question-answer comparisons are insufficient for evaluating quantization quality because flip errors occur when quantization causes models to give opposite answers. Perplexity (PPL) measures how surprised a model is by the next token, but has limitations because it measures average confidence rather than actual behavior. A model might have the same perplexity as the original but still produce different outputs due to changes in the distribution of alternative tokens. Kullback-Leibler Divergence (KLD) measures the difference between probability distributions of original and quantized models, quantifying how much predictions diverge in nats (natural units of information).

Click to watch →

Building Large Language Models: Stanford CS229 Lecture

Building Large Language Models: Stanford CS229 Lecture

1.8M

Perplexity measures model uncertainty as exp(average loss), ranging from 1 (perfect prediction) to vocabulary size. Modern evaluation uses standardized benchmarks like HELM and Hugging Face Leaderboard, which aggregate diverse tasks including question answering. MMLU tests domain-specific knowledge through multiple-choice questions. Evaluation challenges include inconsistent protocols across organizations, chain-of-thought contamination (test data overlapping training data), and difficulty assessing open-ended generation where multiple correct answers exist.

Click to watch →

Run Deepseek locally on a 24g GPU: Quantizing on our Giga Computing 6980P Xeon

Run Deepseek locally on a 24g GPU: Quantizing on our Giga Computing 6980P Xeon

38.3K

Perplexity measures how well a language model predicts the next token in a sequence. It represents the model's ability to generate coherent text. Lower perplexity indicates better performance, meaning the model makes fewer errors in predicting what comes next. When quantizing models, researchers use perplexity to measure how much quality is lost when the model is compressed.

Click to watch →

Deep Dive

Deploying optimized local models using lightweight inference engines like llama.cpp, Ollama, or vLLM.

Llama.cpp vs vLLM: Which Local LLM Engine Actually Scales?

Llama.cpp vs vLLM: Which Local LLM Engine Actually Scales?

429

Running local LLMs provides cost savings, reliability, and privacy benefits. Llama 2 pioneered open-weight models downloadable from Hugging Face. Early models faced hardware constraints with 70B parameter models requiring expensive GPUs. Llama.cpp addressed this through quantization (reducing precision from float16 to int8/int4), GGUF file format combining weights, and CPU inference support. vLLM extends this to production environments with hardware accelerators (Nvidia GPUs, Google TPUs, AMD, Intel). Key optimizations include continuous batching, KV cache management, paged attention, speculative decoding, and disaggregation. Both tools support OpenAI-compatible endpoints, enabling seamless migration from paid APIs to local deployments. llama.cpp targets consumer hardware while vLLM targets production workloads.

Click to watch →

How to Host and Run LLMs Locally with Ollama & llama.cpp

How to Host and Run LLMs Locally with Ollama & llama.cpp

3.4K

Running LLMs locally eliminates API costs and reduces hardware expenses. llama.cpp serves as the foundational C++ library supporting Apple Silicon, NVIDIA GPUs, and other hardware. Ollama simplifies deployment by providing pre-configured models that can be easily pulled and run. Model selection depends on hardware: the Llama 3.1 70B requires 43GB RAM, while laptops with 16GB RAM can typically run models up to 8-12 billion parameters. Llama 3.2 offers small variants (1B and 3B) designed for resource-constrained devices. Model quantization reduces memory usage by representing weights with fewer bits, with Q4 recommended as a balance point. GGUF is the file format used by llama.cpp and Ollama for storing models.

Click to watch →

Ollama vs vLLM vs Llama The ULTIMATE LLM Showdown (2026)

Ollama vs vLLM vs Llama The ULTIMATE LLM Showdown (2026)

337

Local LLM inference engines serve different user needs: Ollama prioritizes simplicity with single-command installation and beginner-friendly setup, vLLM focuses on high-throughput enterprise performance with optimized GPU utilization and memory handling for serving multiple users, while Llama.cpp offers maximum flexibility and hardware compatibility across devices from desktops to phones, allowing advanced users to manually optimize performance and memory usage.

Click to watch →

Qwen 3.5 Vision – The ONLY LOCAL Setup YOU NEED (No Ollama/LM Studio)! It's INSANE!

Qwen 3.5 Vision – The ONLY LOCAL Setup YOU NEED (No Ollama/LM Studio)! It's INSANE!

250

To run Qwen3.5 Vision locally with maximum performance, use llama.cpp for CPU+GPU mixed inference with image support and vLLM for GPU-heavy workloads with video input, both running in Docker containers with proper quantization (FP8 or int4) selected based on your GPU capabilities; this approach provides full parameter control, faster execution than convenience apps like Ollama/LM Studio, and enables applications like OCR, document translation, and video analysis while keeping data private.

Click to watch →

How to Run LLMs Locally (Great For Learning and Privacy)

How to Run LLMs Locally (Great For Learning and Privacy)

5.4K

Modern open-source LLMs like Gwen, Kimmy, and GLM can now run locally on personal devices without requiring hosted APIs, offering privacy benefits and eliminating API costs. Five key tools enable local LLM deployment: llama.cpp (a C++ inference engine supporting CPUs, GPUs, and Apple silicon with GGUF file format and quantization), Ollama (a developer-friendly wrapper handling model downloads and OpenAI-compatible API), LM Studio (a graphical desktop application for casual users), vLLM (a production-grade engine using paged attention and continuous batching for high-throughput serving), and MLX LM (optimized for Apple Silicon with unified memory architecture). The choice of tool depends on use case: Ollama for quick prototyping, LM Studio for casual users, vLLM for production serving, and MLX LM for Apple devices.

Click to watch →

Deep Dive

Investigating complementary model compression and optimization techniques such as structural pruning, knowledge distillation, and Low-Rank Adaptation (LoRA).

What is Low-Rank Adaptation (LoRA) | explained by the inventor

What is Low-Rank Adaptation (LoRA) | explained by the inventor

95.6K

Low-Rank Adaptation (LoRA) is a parameter-efficient method for customizing pre-trained AI models by training only a small number of low-rank matrices (instead of all parameters), which dramatically reduces model checkpoint sizes (e.g., from 1TB to 25MB for GPT-3) while preserving full fine-tuning performance; this approach works by restricting weight matrix updates to a lower-dimensional subspace (rank r < d), storing only 2d×r parameters instead of d², and allows flexible trade-offs between efficiency and expressiveness through two configurable axes: whether to fine-tune all parameters and how expressive the updates should be in terms of matrix rank.

Click to watch →

LoRA explained (and a bit about precision and quantization)

LoRA explained (and a bit about precision and quantization)

118.5K

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique that reduces computational costs when adapting large pre-trained models to new tasks by decomposing weight updates into low-rank matrices (A and B) rather than updating all model parameters, where the change in weights ΔW = BA, and the rank r and scaling factor α control the compression level and adaptation strength respectively; this approach leverages the observation that large language models have low intrinsic dimensionality, meaning only a small subset of parameters needs adjustment to achieve good performance, making it possible to fine-tune massive models (like those with 1.8 trillion parameters) on single GPUs while maintaining efficiency through rank decomposition and optional quantization techniques like QLoRA.

Click to watch →

What is LoRA? Low-Rank Adaptation for finetuning LLMs EXPLAINED

What is LoRA? Low-Rank Adaptation for finetuning LLMs EXPLAINED

96.4K

LoRA (Low-Rank Adaptation) is a parameter-efficient method for fine-tuning large language models that freezes the original model weights and instead adds low-rank matrices (A and B) to represent the weight differences needed for adaptation, reducing the number of trainable parameters from O(d²) to O(rd) where r is the chosen rank, thereby dramatically decreasing computational and memory costs while maintaining effectiveness comparable to full fine-tuning.

Click to watch →

Fine-tuning Large Language Models (LLMs) | w/ Example Code

Fine-tuning Large Language Models (LLMs) | w/ Example Code

551.5K

LoRA is a parameter-efficient fine-tuning method where instead of retraining all weights, you add new trainable parameters (matrices B and A) that are multiplied together to create a low-rank matrix ΔW. The hidden layer becomes W₀X + BA×X, where W₀ is frozen. With d=1000, K=1000, and R=2 (intrinsic rank), LoRA requires only 4,000 trainable parameters versus 1 million for full parameter fine-tuning, achieving massive computational savings.

Click to watch →

Deep Dive: Parameter-Efficient Model Adaptation with LoRA and Spectrum

Deep Dive: Parameter-Efficient Model Adaptation with LoRA and Spectrum

25K

LoRA constrains weight updates by representing them as the product of two low-rank matrices (B × A), where B is n×R and A is R×m, with R typically 4-32. Instead of updating the full weight matrix, LoRA learns these smaller matrices while keeping the base model unchanged. During inference, adapted weights are computed as W₀ + ΔW = W₀ + B × A. This trains only 1% or less of original parameters, primarily applied to attention layers. However, LoRA faces challenges: deciding which layers to apply it to, selecting optimal rank values, and recognizing that the same rank may not work for all layers. Critically, LoRA struggles with pre-training and develops 'intruder dimensions' - new singular vectors not present in the original model, potentially causing catastrophic forgetting. Research shows LoRA performs poorly on math benchmarks like GSM8K, suggesting its approximation harms mathematical reasoning accuracy.

Click to watch →

Quantization Basics

How It Works

Context Memory

Memory Savings

Model Selection

Action Plan

Quantization Basics

0:00

Playing Section

1
Explains need for precision in AI models and RAM cost.

403.3K views14.6K12:09@technovangelistOriginal Release: 2024-12-28

Quantization is a technique that reduces the precision of AI model weights (from 32-bit floating point to lower bit representations like Q2, Q4, or Q8) to significantly decrease memory requirements, enabling large AI models (such as 70B parameter models) to run on standard consumer hardware; Q2 offers the most memory savings but may sacrifice some accuracy, while Q4 provides a balanced trade-off between performance and efficiency, and context quantization further optimizes memory by compressing the conversation history cache.

[00:00:00]this tiny laptop is running a 70 billion parameter AI model right now but wait shouldn't this model be too big to fit the math just doesn't add up unless you know about quantization today I'm going to show you the clever trick that makes this possible and why those mysterious Q2 Q4 and Q8 tags in ama are about to become your best friends in the next 10 minutes or so you'll learn how to on massive AI models on sort of Basic Hardware what Q2 Q4 and Q8 actually mean and it's simpler than you think and which quantization is right for your projects plus I'll show you a brand new trick with context quantization that could save you gigabyt of ram now quick pause if you're serious about running AI models locally hit subscribe and the bell every week I share new tricks like this that can save you hundreds in hard costs and next week well let's just say you'll want to be here for that here's what nobody tells you about AI models they're just giant collections of huge numbers billions of them each one needs to be stored with Incredible Precision think of it like this normally these numbers are stored with 32-bit Precision it's like having a ruler that can measure down to microscopic levels sounds great right but here's the problem all that precision comes at a cost a pretty massive one let me show you exactly what happens to your RAM let's do some quick math take a 7 billion parameter model multiply each of those parameters by four bytes and you need 28 GB just to store it that's more RAM than a lot of gaming PCs have you're looking at$ 2,000 $3,000 for a GPU just to run one model and this model won't even fit in one of those but here's where quantization changes everything think of quantization like choosing different rulers maybe the full model is using millimeters Q8 is measuring in centim less precise but still pretty accurate Q4 now we've got a marker every 5 cm or so but look how much extra space we have and Q2 well this is the extreme option like measuring with a random stick from your yard but sometimes that's all you need here's my favorite way to explain this imagine you're running a mail room but instead of postcards and packages you're soaring numbers in the original unquantized model every number gets its own custom siiz mailbox it's perfect organization but imagine how much space that takes up that's exactly what happens with full 32-bit Precision it's like having an infinite mail room where every tiny value gets its own special spot here's what regular Q4 Quan ation looks like in action imagine having just 16 mailboxes for all your numbers every number has to pick the closest available slot it works but it's not very smart what happens when you have two big bunches of numbers one bunch of really tiny numbers and the other bunch of really huge ones but wait it gets better see those models tagged with K and olama they use something called kqu and they are brilliant instead of one rigid system kqu create multiple specialized mail rooms small numbers get their own precise area and big numbers get their space it's like having a smart assistant organizing everything perfectly the different sizes you see KS km and KL for small medium and large are different levels of detail in the notes that are in the mail room the small version keeps brief notes medium keeps more detailed records and large maintains very precise information about how everything was arranged this smart system means areas with mostly small numbers get more precise small slots while areas with bigger numbers get appropriately sized spaces it's like having a mail room that adapts to what you're storing rather than forcing everything into the same size box but here's what really matters how this affects your actual usage look at memory watch what happens when we load a Q4 model now check out speed noce the startup time and performance let's run the same prompt on different versions now here's where it gets really interesting AMA just dropped a gamechanging feature context quantization see we've been talking about making the models smaller but there's another memory hog we haven't discussed yet your conversation history a year ago this wasn't really a big deal models could only remember 2,000 to 8,000 tokens of conversation but now well we're talking about models that can remember entire books worth of conversation that's 128k or 128,000 or so tokens or more here's the problem all that conversation history eats up Ram but context quantization fixes this let me show you how to enable this it's super simple first we'll turn on flash attention by setting olama flash attention is true and then set the cache type with olama KV cach type F16 what what happens to the memory usage when we enable these so let's get practical I'm going to show you exactly how much memory you can save with these tricks for this demo I'm using quen 2.5 a 7 billion parameter model out of the box it uses Q4 km quantization and takes up 4.7 GB on dis first let's max out the contact size by default olama gives us 2K tokens or 240 8 tokens but quen can handle 32k or 32,768 here's an easy way to set that up first go to ama.com and find the model then run the command shown on the right now type /set parameter numor CX 32768 and hit enter and then type save Quin 2.5 Max or whatever you want to call it and hit enter finally quit olama now if if you run AMA LS you should see that the model has been added to your list of models perfect we have our maxed out model ready to go now I'll run the model and watch the output from ASI top which is a similar tool to NV toop on Linux but for Mac OS before I hit enter on the command which is this I see the memory usage hanging out around 25 gigs the Mac uses unified memory so that's the total amount of memory used right now now by the system and the GPU now I hit enter and I see that memory usage climbs to around 40.9 gbes so this model with a 32k context uses about 15 GB now I'll quit ol serve and run this command which runs olama serve again but this time with the flash attention option enabled I'm doing this at the command line to easily switch back and forth between settings rather than setting up the environment variables the correct way I'll run the model with the same question and this time I see the memory top out at 33.7 gbt that's a massive drop in memory usage from 40.9 GB that's a savings of about 7 gigs let's do this one more time setting the KV cache quantization to Q8 Now it only climbs to 30.6 GB that means it's taking up closer to 5 gigs of memory with the maxed out contact size let's try this okay one more time going back to the original model with the default contact size of 2K without using flash attention at all now it goes from 28.5 to 30.3 so with a normal contact size it needs less than 2 GB the maxed out context needs 15 gigs and using context quantization for this model set to Q8 saves US 10 gigs bringing it down to only 5 gigs but it's important to note that not every model is going to benefit in the same way I saw one model take less memory of flash attention on but take more memory with KB cach quantization set to Q8 more than without using flash attention at all that was a brand new model by IBM so it's not always guaranteed to use Flash attention and context quantization hopefully now you have a better understanding of how quantization works and how it affects the performance and efficiency of the model and its context well the next question is probably going to be how to pick the right model for your specific needs in general it's usually best to start out with one of the Q4 models this could be the Q4 or even Q4 km which is what AMA seems to be defaulting to these days if you notice issues with the generation quality you can move up to Q8 or even fp16 if it seems to work well you can try dropping further down to Q2 it's amazing how often Q2 model will work just as well for most tasks for most people and the memory usage is so much lower combine that with a quantized context and you're getting a lot of power on a machine with even the most limited memory now let's wrap this up with what really matters whether you're running AI on a Raspberry Pi or a beast of a workstation here's your game plan start simple begin with a Q4 km model and then enable flash attention and test your specific use case then optimize it a bit if that's running smooth try Q2 but if it's having issues move to Q8 need more context try playing with Q8 KV cache quantization the best part these tricks can turn a large model into something that runs on a mostly normal laptop that's not just Theory you saw it happen right here quick action steps download a AMA Q4 model enable flash attention and test with your use case experiment with lower quantization you can join our Discord or theama Discord for more optimization tips and don't forget to like And subscribe for next week's Deep dive into something else remember the perfect setup isn't about using the highest settings it's about finding what works for your specific needs thanks so much for watching and I look forward to seeing you in the next video goodbye are uh something for a video are you wasting p a little bit Jase Jase one two three four five Dad Dad this is emergency T I know it's it's being used for emergency no it is not dad you are not Hur and you are wasting tape that we might need for something medical my video are emergencies no they're not okay this is my last one

Up Next

Howie AI: Austin Petersmith on Building an AI Secretary

@ThePeelPod

586 views•2025-10-09

Building Real-Time ML Pipelines with Feature Stores and MLOps Frameworks

@ODSCAI

5.1K views•2022-02-20

Bypassing Tor Censorship: Bridges and Pluggable Transport Guide

@Coding_ForEveryone

397 views•2024-06-11

Neural Networks Explained: Math, Layers, and Learning Fundamentals

@3blue1brown

21.9M views•2017-10-05

Related Study Plans & Knowledge Roadmaps

Structured learning paths in Artificial Intelligence

View All Artificial Intelligence Topics

artificial-intelligence

Building Recommenders: Two-Tower & Vector DBs

01Foundations of Python & Machine Learning Basics

02Introduction to Recommender Systems

03Deep Learning & Representation Learning

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

Latent Diffusion: PyTorch, UNet & CLIP

01Deep Learning & PyTorch Foundations

02Convolutional Networks & Autoencoders

03The Math and Mechanics of Diffusion Models

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

NeRFs: 3D Reconstruction & Instant-NGP

01Math & Python Foundations for 3D Vision

02Photogrammetry & Camera Pose Estimation

03Neural Radiance Fields (NeRF) Core Concepts

+ 2 more learning modules

Explore Roadmap

artificial-intelligence

TFT Forecasting: Energy Demand & Deep Learning

01Programming & Data Science Foundations

02Introduction to Machine Learning & Time Series

03Deep Learning & Sequential Models

+ 3 more learning modules

Explore Roadmap