Academic Databases
Explore Our Semantic Databases
STEM & Tech
Math, Physics, CS & Engineering...
Economics
Macro, Micro, Econometrics...
Business
Management, Marketing, Finance...
Health & Life Sciences
Biology, Medicine, Neuroscience...
Humanities & Social Sciences
Psychology, Law, Philosophy, History...
General Studies
General & interdisciplinary studies...
Explore Topics
Features

Fine-Tuning Llama 3.2 with Unsloth for Local Inference via Ollama

Added: 2026-08-06

Deep Dive

Foundational understanding of Large Language Models (LLMs), including the transformer architecture, tokenization, and autoregressive text generation.

Por dentro das LLMs como ChatGPT - O Curso Completo

Por dentro das LLMs como ChatGPT - O Curso Completo

215K

Large Language Models (LLMs) are AI systems that can converse about any topic, execute tasks, and create documents. They belong to Natural Language Processing (NLP), which studies human language. The Transformer architecture, proposed by Google Brain in 2017, revolutionized NLP by outperforming previous Recurrent Neural Networks (RNNs). Training requires massive datasets from CommonCrawl, a non-profit that crawls the entire internet. The data cleaning process involves: (1) RL filtering to remove sensitive content; (2) Text extraction to remove HTML tags; (3) Language filtering; (4) Privacy removal. Tokenization transforms text into tokens through UTF encoding, byte grouping, and Byte Pair Encoding (BPE), making processing more efficient. Transformers measure statistical relationships between token sequences to generate text by predicting the most likely next token at each step. LLMs generate text token by token through an autoregressive process: input text is converted to tokens, fed into the model, which outputs probability distributions over all possible next tokens, and a random selection picks the next token. Training involves billions of parameters (e.g., ChatGPT-3.5 has 175 billion) that are adjusted to improve prediction accuracy. A base model is a neural network trained on internet data that can generate text but lacks assistant capabilities. In-context learning allows models to understand patterns from examples provided in the prompt.

Click to watch →

Low Level Technicals of LLMs: Daniel Han

Low Level Technicals of LLMs: Daniel Han

53.6K

This section covers Transformer architecture fundamentals and tokenization. The speaker explains that the Transformer is the foundation behind all modern language models (Gemma, Llama, Mistral, etc.), designed for sequence modeling and applicable to any sequential data. The architecture consists of stacked layers with attention mechanisms and feed-forward networks. The speaker introduces Transformer Plus+, an enhanced version with RoPE embeddings, SwiGLU activation, RMSNorm, and untied weights, considered the best current Transformer architecture. The section details Transformer layer components: LayerNorm, RoPE embeddings, attention mechanism, residual connections, and another LayerNorm, repeated L times (e.g., 32 layers in Llama 3). The speaker then covers tokenization, explaining that computers only understand numbers, so text must be converted to numerical IDs. The speaker demonstrates creating a simple tokenizer by combining punctuation with words and ignoring spaces, then discusses vocabulary size, word stemming, and lowercasing as techniques to reduce vocabulary size.

Click to watch →

Explained: The conspiracy to make AI seem harder than it is! By Gustav Söderström

Explained: The conspiracy to make AI seem harder than it is! By Gustav Söderström

375K

The Transformer architecture, introduced in 2017, revolutionized language modeling by solving the context scalability problem. Its attention mechanism dynamically assigns different weights to different words, preventing computational complexity from exploding with longer contexts. This enables handling thousands to tens of thousands of words simultaneously. Text generation follows an autoregressive process where the model predicts one word at a time, feeding its own output back into itself iteratively to build complete sentences.

Click to watch →

Large Language Models (LLMs) - Everything You NEED To Know

Large Language Models (LLMs) - Everything You NEED To Know

435K

LLMs operate through three core processes: tokenization splits text into ~34-character units; embeddings convert tokens into numerical vectors capturing semantic meaning and relationships; and Transformers use multihead attention to understand word context through dot product calculations. Words frequently appearing together receive similar embeddings, enabling next-word prediction. This architecture transforms input matrices into output matrices corresponding to predicted words, with transformations determined during training based on learned word sequence patterns.

Click to watch →

Partial Objects: Yeah, stop telling people we don't understand LLM's and AI.

Partial Objects: Yeah, stop telling people we don't understand LLM's and AI.

7.3K

The transformer model architecture, introduced in the 2017 paper 'Attention is All You Need' from Google, is the foundation for almost all major language models today. This architecture allows words to assign different levels of importance to other words in a sentence. LLMs do not work with actual words—they work with tokens, which are numerical representations of words or parts of words. Understanding tokenization is crucial because it explains why LLMs behave the way they do.

Click to watch →

Deep Dive

The conceptual difference between base model pre-training and supervised fine-tuning (SFT) for adapting models to specific downstream tasks.

What is Difference Between Pretraining and Finetuning?

What is Difference Between Pretraining and Finetuning?

2.6K

Pretraining is the process of training a base model with a large, generic dataset to learn general patterns and set initial parameters, while finetuning is the subsequent process of adapting a pre-trained model to a specific task by training only a section of the model using the organization's own data, leveraging the existing parameter weights for efficient learning.

Click to watch →

Stanford CS336 Language Modeling from Scratch | Spring 2025 | Lecture 1: Overview and Tokenization

Stanford CS336 Language Modeling from Scratch | Spring 2025 | Lecture 1: Overview and Tokenization

462.3K

Alignment makes base language models useful for specific applications. Supervised fine-tuning (SFT) uses prompt-response pairs to train models to follow instructions. Remarkably, even 1,000 examples can give instruction-following capabilities to a good base model. SFT is similar to pre-training - you maximize the probability of the text given the prompt. This phase transforms a base model capable of next-token prediction into an aligned model that follows user instructions.

Click to watch →

Stable Diffusion Models Explained Once and for All (1.5, 2, XL, Cascade, 3)

Stable Diffusion Models Explained Once and for All (1.5, 2, XL, Cascade, 3)

79.3K

A base model refers to the fundamental architecture and training approach of a model family. Fine-tunes are specialized versions created by taking a base model and training it further on specific datasets or aesthetics. For example, NovelAI models are considered fine-tunes branching off from the mainline Stable Diffusion 1 trunk, specializing in anime aesthetics while the trunk remains a general-purpose model.

Click to watch →

Basta confusione: i concetti principali dell’AI spiegati in breve

Basta confusione: i concetti principali dell’AI spiegati in breve

34.8K

Supervised Fine-Tuning is the second phase of LLM training where the model's parameters are further refined to solve specific tasks. Unlike pretraining which uses massive datasets, SFT uses smaller but carefully curated datasets where humans provide the correct answers. This phase teaches the model to follow instructions and perform specific tasks like sentiment analysis, question answering, or code generation, making the model more useful for practical applications.

Click to watch →

Por dentro das LLMs como ChatGPT - O Curso Completo

Por dentro das LLMs como ChatGPT - O Curso Completo

215K

Supervised Fine-Tuning transforms base models into assistants by training them on synthetic conversation data. OpenAI created new tokens (IMSTART, IMSEP, IMEND) to structure conversations between users and assistants. A team of 40 freelancers created examples of ideal assistant responses. The model learns to recognize these conversation patterns and generate appropriate responses. This stage also defines the model's personality and behavior style.

Click to watch →

Deep Dive

Core principles of Parameter-Efficient Fine-Tuning (PEFT), specifically how Low-Rank Adaptation (LoRA) optimizes only a fraction of model parameters to save VRAM.

Fine-Tuning Large Language Models Explained | Types, Use Cases & Complete Guide

Fine-Tuning Large Language Models Explained | Types, Use Cases & Complete Guide

259

PEFT addresses full fine-tuning drawbacks by training only a small percentage of parameters while keeping most frozen. The trainable parameters are called 'adapters.' For a 13-billion parameter LLM, full fine-tuning requires hundreds of GBs of VRAM, while PEFT fits on a single 24GB GPU. LoRA (Low-Rank Adaptation) injects low-rank matrices into weight layers, training only these matrices while base weights remain frozen. For a 7-billion parameter Llama model fine-tuned on legal documents, LoRA requires only 40MB of trainable parameters versus 30GB for full fine-tuning.

Click to watch →

Parameter Efficient Fine Tuning

Parameter Efficient Fine Tuning

849

LoRA decomposes weight updates into two low-rank matrices A and B (W' = W + A×B), freezing original weights while training only A and B. This reduces trainable parameters from d² to 2rd, where r is typically 1-2. LoRA is typically applied to attention layers but not MLP layers. Despite using less than 1% of parameters, LoRA achieves performance comparable to or better than full fine-tuning. The method works because original weights capture most expressive capacity, and small rank-1 changes can significantly adjust model behavior.

Click to watch →

Fine-Tuning Local Models with LoRA in Python (Theory & Code)

Fine-Tuning Local Models with LoRA in Python (Theory & Code)

17.8K

LoRA (Low Rank Adaptation) is a parameter-efficient technique for fine-tuning pre-trained neural networks. The core mathematical insight is that when adapting a pre-trained model to a new task, the changes needed to the weight matrix have lower intrinsic rank than the full matrix. Instead of training all parameters, LoRA decomposes the weight update into two small matrices B and A (with rank R << original dimensions), reducing training parameters from D_out×D_in to D_out×R + R×D_in. This enables fine-tuning on consumer GPUs with limited VRAM while maintaining effectiveness. The complete Python implementation pipeline involves: installing dependencies (PyTorch, HuggingFace Transformers, Datasets, PEFT, BitsAndBytes), loading a pre-trained model with 4-bit quantization, configuring LoRA parameters (rank, scaling factor, target modules, dropout, task type), preparing training data through tokenization, setting up training arguments, and executing the training loop using the Trainer class.

Click to watch →

What is Low-Rank Adaptation (LoRA) | explained by the inventor

What is Low-Rank Adaptation (LoRA) | explained by the inventor

95.6K

LoRA offers practical advantages over alternative parameter-efficient methods like prefix tuning or adapters. Since full fine-tuning is a special case of LoRA, practitioners can start with low-rank configurations and progressively increase rank if performance is insufficient, providing clear progression toward full fine-tuning. LoRA applies to any model using matrix multiplication, including WaveNet and support vector machines. During inference, LoRA introduces no additional latency because updates are additive and can be pre-multiplied into the base parameters. Rapid model switching is enabled by subtracting updates to recover the original model and load new modules, with the process faster than a single forward pass. Engineering optimizations include caching thousands of LoRA modules in RAM (since RAM exceeds VRAM capacity), enabling parallel training of multiple modules on different tasks using shared base weights, and hierarchical specialization where models are adapted gradually through successive stages with varying ranks.

Click to watch →

LLM Model Fine Tuning Simplistically Explained

LLM Model Fine Tuning Simplistically Explained

287

Parameter-Efficient Fine-Tuning (PEFT) addresses the computational expense of full model fine-tuning, which requires approximately four times more GPU resources than hosting the model for inference. LoRA (Low-Rank Adaptation) is the most common PEFT method, adding only a small number of parameters (e.g., 1 million to a 150 million parameter model) and fine-tuning only those additions while keeping original weights frozen. This approach significantly reduces computational costs while maintaining the ability to adapt pre-trained models to new tasks.

Click to watch →

Deep Dive

Basic proficiency in Python, PyTorch, and navigating the Hugging Face ecosystem, particularly using the 'transformers' and 'datasets' libraries.

Hugging Face : La bibliothèque IA qui change tout !

Hugging Face : La bibliothèque IA qui change tout !

10.6K

Hugging Face offers various assistants for AI tasks including image generation, video generation, audio generation, music generation, code generation, and document analysis. Spaces functions as an App Store for AI applications, allowing users to search for specific tasks (face swap, image editing, video generation) and find corresponding models. Some Spaces are marked as 'running on zero,' providing free access without GPU consumption. Transformers is a library grouping numerous Large Language Models for specific tasks, with Python code provided for local integration. Datasets provide databases for training machine learning models, including time series, text, video, and audio data, filterable by format (CSV, parquet) and data type. The platform offers both free and Pro versions (9 dollars/month) with community documentation and support.

Click to watch →

Stanford CS224N NLP with Deep Learning | 2023 | Hugging Face Tutorial, Eric Frankel

Stanford CS224N NLP with Deep Learning | 2023 | Hugging Face Tutorial, Eric Frankel

65.5K

This tutorial demonstrates how to use the Hugging Face Transformers library for natural language processing tasks, covering the complete workflow from installing the library and finding pre-trained models on the Hugging Face Hub, through tokenizing input text using AutoTokenizer, loading pre-trained models with AutoModelForSequenceClassification, preparing datasets with the Datasets library, and implementing both manual PyTorch training loops and the simplified Trainer class for model fine-tuning on tasks like sentiment analysis.

Click to watch →

‫آموزش ترنسفورمرز هاگینگ فیس (Transformers Hugging Face): فراخوانی مدل

‫آموزش ترنسفورمرز هاگینگ فیس (Transformers Hugging Face): فراخوانی مدل

361

This comprehensive section covers the foundational knowledge for working with Hugging Face AI models in Python. It introduces Hugging Face as a platform for accessing state-of-the-art AI models and explains the Transformers library for loading them. The video covers installing Transformers (pip install transformers for local, pre-installed in Colab), exploring Hugging Face libraries (Transformers, Datasets, Tokenizers, Sentence Transformers), and understanding the two main model loading methods: Pipeline (simpler, recommended for beginners) and AutoClass (more detailed, more code). The section demonstrates practical implementation using Koala model, showing how to select models by category and parameter count, create pipelines with task='text-generation', and pass input text. It covers different task types supported by Pipeline including audio-classification, depth-estimation, image-text-to-text, and masked-language-modeling. The video emphasizes hands-on practice with exercises for testing with ModernBERT and other models.

Click to watch →

Attention & Co- Getting started with Transformers

Attention & Co- Getting started with Transformers

2.2K

Hugging Face is an open-source company dedicated to building a community for AI, making state-of-the-art machine learning accessible to everyone. Their ecosystem includes: (1) The Hub - a central repository with 22,000+ models, datasets, and tools, (2) Transformers library - unified API supporting 50+ architectures, (3) Datasets library - optimized for processing large datasets, (4) Accelerate - simplifies multi-GPU training, (5) Spaces - platform for building web applications to demonstrate models. The Transformers library provides a Pipeline API that simplifies common NLP tasks like sentiment analysis, question answering, and translation. Training transformers can be done with just a few lines of code using the Trainer class. The library supports many different architectures using a unified API, allowing practitioners to experiment with different models by simply changing the model name in the code.

Click to watch →

PyTorch Crash Course: Building Neural Networks in Python

PyTorch Crash Course: Building Neural Networks in Python

PyTorch provides built-in datasets (torchvision.datasets) and data loaders (torch.utils.data.DataLoader) for efficient data handling. Datasets store data samples, while data loaders provide optimized iteration with batching, shuffling, and parallel loading. Transformations (torchvision.transforms) can be applied to preprocess data. Example: torchvision.datasets.MNIST with transforms.ToTensor() converts images to tensors.

Click to watch →

Unsloth Setup

Dataset Prep

LoRA Setup

Format Data

Train Model

Training Eval

Export GGUF

Ollama Run

Local Test

Unsloth Setup

0:00

Playing Section

1
Identify 1B and 3B models as suitable for on-device runs.
2
Install the nightly version of Unsloth for fine-tuning.
3
Load the 3B instruct model with 4-bit quantization.

Deep Dive

Advanced alignment methodologies such as Direct Preference Optimization (DPO) and Reinforcement Learning from Human Feedback (RLHF) to align models with human preferences.

Direct Preference Optimization: Forget RLHF (PPO)

Direct Preference Optimization: Forget RLHF (PPO)

16.1K

Direct Preference Optimization (DPO) is a new training methodology developed by Stanford University that bypasses the complex two-step process of Reinforcement Learning from Human Feedback (RLHF) by directly optimizing language models using a single maximum likelihood objective based on human preference data, achieving similar or better performance than traditional RLHF methods while being substantially simpler to implement and train.

Click to watch →

Reinforcement Learning, RLHF, & DPO Explained

Reinforcement Learning, RLHF, & DPO Explained

15.4K

This video explains three optimization techniques for aligning AI language models with human preferences: (1) RLHF uses reinforcement learning where a reward model scores outputs and an optimizer updates the policy model through trial and error, but faces challenges with instability and expensive human labeling; (2) DPO directly optimizes the language model on preference data without explicit reward modeling, achieving equivalent or better results than RLHF while being simpler, faster, and more robust to temperature changes; (3) KTO applies prospect theory from behavioral economics to maximize utility by considering human risk aversion, using a sigmoid value function and handling noisy/inconsistent data better than other methods, with tunable hyperparameters for balancing risk aversion between desirable and undesirable examples.

Click to watch →

Aligning LLMs with Direct Preference Optimization

Aligning LLMs with Direct Preference Optimization

33.9K

Direct Preference Optimization (DPO) is a simplified alignment technique that trains language models directly on human-preference data without requiring a separate reward model, using a differentiable loss function that maximizes the difference between log probabilities of preferred versus dispreferred responses; this approach achieves comparable or better results than traditional reinforcement learning from human feedback (RLHF) while being more computationally efficient and easier to implement, as demonstrated by its successful application in training Zephyr, a state-of-the-art 7B parameter chat model.

Click to watch →

CS 285: Eric Mitchell: Reinforcement Learning from Human Feedback: Algorithms & Applications

CS 285: Eric Mitchell: Reinforcement Learning from Human Feedback: Algorithms & Applications

8.5K

Reinforcement Learning from Human Feedback (RLHF) is a four-step pipeline for training large language models: (1) unsupervised pre-training on internet text, (2) supervised fine-tuning on human demonstrations, (3) fitting a reward model using preference data collected from human comparisons of model responses, and (4) reinforcement learning with the reward model and KL-divergence constraint to optimize the policy. Direct Preference Optimization (DPO) simplifies this process by deriving a closed-form optimal policy directly from preference data, eliminating the need for separate reward model training and RL optimization loops. DPO achieves this by parameterizing the reward function as the log probability ratio between the current policy and a reference model, transforming the RLHF objective into a tractable classification loss that can be optimized directly without rollouts.

Click to watch →

George Hotz | Reading & Talking | let's read ML papers | from $1499 buy comma.ai/shop/comma-three

George Hotz | Reading & Talking | let's read ML papers | from $1499 buy comma.ai/shop/comma-three

80.9K

RLHF (Reinforcement Learning from Human Feedback) is a technique for aligning language models with human preferences. It involves training a reward model that takes text sequences and returns a scalar reward representing human preference. Companies like Midjourney use this approach where users select preferred outputs, and this preference data is used to optimize models. This allows gathering human preference data without encoding it in large datasets.

Click to watch →

Deep Dive

Deep dive into model quantization formats (such as GGUF and AWQ) to optimize memory footprint and execution speed on resource-constrained consumer hardware.

LLM Quantization Explained

LLM Quantization Explained

389

LLM quantization is a technique that reduces the memory footprint of large language models by storing their parameters in fewer bits (such as FP16, BF16, FP8, INT8, or INT4), enabling models like the 70B Llama 3 to run on consumer hardware that would otherwise require 140GB of memory; this process involves three key components—number formats determining bit precision, quantization methods like GPTQ and AWQ that preserve model quality by identifying important weights, and file formats like GGUF that store the compressed model for local execution—though lower precision introduces trade-offs where models may degrade in reasoning, coding, and instruction-following capabilities.

Click to watch →

Which Quantization Method is Right for You? (GPTQ vs. GGUF vs. AWQ)

Which Quantization Method is Right for You? (GPTQ vs. GGUF vs. AWQ)

37.7K

This tutorial explains three common quantization methods for running large language models on limited hardware: GPTQ (post-training quantization optimized for GPU inference), GGUF (formerly GGML, focused on CPU execution with optional GPU layers), and AWQ (activation-aware weight quantization that prioritizes important weights). The choice depends on hardware constraints—use native loading if GPU capacity permits, GPTQ for moderate GPUs, GGUF for very small GPUs, and AWQ for newer applications where available.

Click to watch →

meu SaaS depois de 6 meses

meu SaaS depois de 6 meses

24.9K

GGUF (GPT Generated Unified Format) is a standardized format for storing quantized AI models. Quantization is the process of reducing model precision by converting weights from float32 to lower precision formats like float16, Q2, Q3, or Q4. This technique trades some model quality for significant improvements in memory usage and inference speed. For example, a 7-billion parameter model that normally requires 28GB of RAM can be compressed to run in just 4GB of RAM using quantization.

Click to watch →

Quantizing LLMs - How & Why (8-Bit, 4-Bit, GGUF & More)

Quantizing LLMs - How & Why (8-Bit, 4-Bit, GGUF & More)

22.8K

GGUF (GPT-Quantized Unified Format) enables efficient cross-platform deployment of quantized language models through llama.cpp framework. This format consolidates entire models into single files, eliminating complex directory structures. Supported quantization levels range from 8-bit down to 2-bit, with methods like Q4_K_M applying 6-bit quantization to attention layers and 4-bit to feed-forward layers. The conversion process involves downloading models from Hugging Face, converting to GGUF format, and applying quantization through command-line tools. Resulting models load directly into system RAM (~800MB for 1B parameter models) and run on CPUs without GPU requirements, demonstrating how quantization enables practical AI deployment on consumer hardware.

Click to watch →

Which .GGUF Should You Download? (Hugging Face Quantization Guide)

Which .GGUF Should You Download? (Hugging Face Quantization Guide)

1.5K

Model quantization trades memory for fidelity by reducing bits per weight, creating a practical ladder where each step down saves memory but costs quality. GGUF serves local tools (Ollama, LM Studio) with download-and-go simplicity; GPTQ/AWQ serve PyTorch environments with different compression approaches; QAT trains models to maintain BF16-like behavior at tiny footprints. K versions outperform older numbering systems. Critical traps include selecting base vs instruct builds and underestimating mixture-of-experts model sizes. Context length requirements drive quantization choices: 32K+ needs Q6/Q8 or QAT. Future directions include KV cache compression for long context and BitNet training with three-value parameters. Practical defaults: GGUF with Q5_K_M, escalate to Q6/Q8 for long context, and adopt QAT when memory constraints demand it.

Click to watch →

Deep Dive

Integrating local Ollama-served models into application development frameworks like LangChain or LlamaIndex to build Retrieval-Augmented Generation (RAG) pipelines.

Ollama meets LangChain

Ollama meets LangChain

59.6K

This tutorial demonstrates how to run large language models locally using Ollama and integrate them with LangChain for various tasks. The video covers three main examples: loading Ollama models (like LLaMA-2) using LangChain's pre-made LLM wrapper with streaming callbacks, creating basic chains with prompt templates and parameters like temperature and max tokens, and building a RAG (Retrieval Augmented Generation) system that scrapes web pages, splits content, stores it in ChromaDB vector database, and uses the LLM to answer questions based on the retrieved information. This enables developers to perform tasks like extracting headlines from websites without writing custom parsing code, as the language model handles the information extraction automatically.

Click to watch →

Python AI Web Scraper Tutorial - Use AI To Scrape ANYTHING

Python AI Web Scraper Tutorial - Use AI To Scrape ANYTHING

346.3K

Ollama is a tool that allows running open-source large language models locally without API tokens. LangChain provides integration between Python code and various LLMs. To use them together: (1) install Ollama and pull a model (e.g., llama3), (2) import from langchain_community.llms import OllamaLLM, (3) create a model instance specifying the model name, (4) define a prompt template containing variables for Dom content and parse description, (5) create a chain by piping the prompt to the model, then invoke the chain with specific content and description inputs.

Click to watch →

I Built a ChatGPT for My Own Documents - LangChain + Ollama (100% Free & Local)

I Built a ChatGPT for My Own Documents - LangChain + Ollama (100% Free & Local)

244

A Retrieval Augmented Generation (RAG) pipeline combines document retrieval, context augmentation, and LLM generation to answer questions based on custom documents without retraining the model. The pipeline works by: (1) loading documents and splitting them into chunks, (2) converting chunks to vector embeddings using models like nomic-embed-text, (3) storing embeddings in a vector database like ChromaDB, (4) retrieving relevant chunks when a query is asked, (5) combining retrieved chunks with the query into a prompt, and (6) generating an answer using a local LLM like Ollama's Llama 3.1. This approach enables accurate, context-grounded responses while maintaining 100% privacy by running entirely locally without API keys.

Click to watch →

100% Local RAG with DeepSeek-R1, Ollama and LangChain - Build Document AI for Your Private Files

100% Local RAG with DeepSeek-R1, Ollama and LangChain - Build Document AI for Your Private Files

26.4K

This video tutorial demonstrates how to build a local Retrieval-Augmented Generation (RAG) document AI assistant using open-source tools: DeepSeek-R1 as the language model, Ollama for local model hosting, and LangChain for building the RAG pipeline. The system processes uploaded documents by chunking them into smaller pieces, enhancing chunks with contextual information, and using a hybrid retrieval approach combining semantic search (via embeddings) with BM25 keyword search. The retrieved context is then passed to the language model along with chat history to generate answers. The complete application is built with Streamlit for the user interface, enabling users to upload files, chat with their documents, and see relevant sources cited in responses.

Click to watch →

Build a Talking Local RAG Stock Advisor (ElevenLabs, Ollama, LangChain, ChromaDB )

Build a Talking Local RAG Stock Advisor (ElevenLabs, Ollama, LangChain, ChromaDB )

26.6K

This video demonstrates how to build a fully local Retrieval Augmented Generation (RAG) pipeline for stock analysis using Llama 3, Ollama, LangChain, and ChromaDB. The key steps include: (1) parsing complex PDFs using PDFPlumberLoader instead of PyPDFLoader for better table and graph handling, (2) splitting text with optimal chunk size and overlap using RecursiveCharacterTextSplitter, (3) generating high-quality embeddings using Ollama with Nomic embedding model for superior results compared to fast embedding, (4) storing embeddings in ChromaDB with persistence optimization, (5) implementing multi-query retriever with Llama 3 via Ollama to generate multiple question variations, and (6) using ElevenLabs for text-to-speech audio generation. The complete pipeline enables building a talking stock advisor that provides evidence-based financial analysis based on document content.

Click to watch →

Deep Dive

Systematic evaluation of fine-tuned models using benchmarking suites (e.g., LM-Eval-Harness) to quantify performance gains and detect catastrophic forgetting.

Fine-Tuning Mistral 7B using QLoRA and PEFT on Unstructured Scraped Text Data | Making it Evil?

Fine-Tuning Mistral 7B using QLoRA and PEFT on Unstructured Scraped Text Data | Making it Evil?

5.8K

The LM Evaluation Harness provides standardized testing across over 200 metrics, powering the Hugging Face LLM Leaderboard. Task selection (arithmetic tests, truthfulness tests) enables rapid assessment of catastrophic forgetting in fine-tuned models within 30 minutes. A critical research question addresses whether fine-tuning can genuinely expand model knowledge. Training GPT-J6B on 1.1GB of Kiwi Farms posts demonstrated that base models ignorant of internet figures could acquire descriptive capabilities post-training, suggesting new knowledge acquisition is possible. However, models deliver inaccurate information with the same confidence as facts, highlighting fundamental limitations in veracity verification.

Click to watch →

Fine-Tuning Your Own Llama 3 Model

Fine-Tuning Your Own Llama 3 Model

7.4K

Model evaluation depends on the fine-tuning goal: general-purpose models use benchmarks like Open LLM Leaderboard or LM Evaluation Harness; domain-specific models (legal, financial, medical) require custom benchmarks; task-specific models (summarization, information extraction) need specialized evaluation. Custom benchmarks can be created for unique use cases, especially important for non-English models.

Click to watch →

AI on Demand 1st Workshop Innovation benchmarking initiatives

AI on Demand 1st Workshop Innovation benchmarking initiatives

117

Benchmarking suites address the problem of comparing machine learning results across papers, which is difficult because authors use different datasets, preprocessing methods, train-test splits, and evaluation measures. Creating a benchmarking suite involves defining constraints (data size, feature types, categorical vs numeric, artificial vs real data) and having the system identify compliant datasets. Once created, any researcher can download the suite and test their models across all included datasets, automatically sharing results back to the platform. This creates an ever-growing knowledge base of how different models perform across various problem types, enabling systematic comparison and identification of model strengths and weaknesses.

Click to watch →

Catastrophic Forgetting can NOT be avoided in Finetuned LLMs | New LLMs Paper

Catastrophic Forgetting can NOT be avoided in Finetuned LLMs | New LLMs Paper

875

Catastrophic forgetting in fine-tuned large language models depends primarily on fine-tuning loss, following an inverse linear relationship where better performance on new tasks leads to greater forgetting of original knowledge; this forgetting increases according to a power law with both the number of parameters fine-tuned and the number of update steps, making it unavoidable through early stopping or parameter reduction.

Click to watch →

Finetuning a Large Language Model on a Custom Dataset

Finetuning a Large Language Model on a Custom Dataset

1.1K

After fine-tuning, models can be evaluated on benchmark tasks like TruthfulQA, BullQA, and others using evaluation scripts. LIT integrates LM Eval Harness from Hugging Face and HELM from Stanford, providing over 100 evaluation tasks. Results are saved to JSON files, allowing comparison between the fine-tuned model and the original pre-trained model.

Click to watch →

Deep Dive

Transitioning from local hobbyist deployment to production-grade serving using high-throughput LLM engines like vLLM or TGI (Text Generation Inference).

Fast LLM Serving with vLLM and PagedAttention

Fast LLM Serving with vLLM and PagedAttention

55.5K

vLLM achieves 24x higher throughput compared to naive Hugging Face model.generate usage and up to 3.5x higher throughput compared to Hugging Face Text Generation Inference (TGI). The project supports approximately 95% of popular LLMs and has been deployed in production environments serving thousands of daily conversations. Real-world deployment reduced the total number of GPUs required by 50% while achieving 2-3x more requests per second. The project has gained widespread adoption across industry and academia.

Click to watch →

I Tested All 4 LLM Deployment Methods So You Don't Have To | Ollama, LLama.cpp, LM studio, vLLM

I Tested All 4 LLM Deployment Methods So You Don't Have To | Ollama, LLama.cpp, LM studio, vLLM

103

vLLM enables production-grade LLM deployment handling multiple concurrent API requests efficiently. Deployment requires creating GPU instances (A100 with 80GB VRAM), installing via pip, and resolving dependency conflicts. Commands specify model IDs, hosts, ports, context sizes, GPU counts, and model parameters. vLLM optimizes for commercial applications serving multiple users simultaneously. Framework selection depends on use case: LM Studio suits casual users with GUI simplicity; llama.cpp offers developer flexibility; vLLM handles production-scale serving. Most individual users should choose between LM Studio for ease or llama.cpp for customization.

Click to watch →

Running vLLM on Akash | Akash Tutorials

Running vLLM on Akash | Akash Tutorials

626

vLLM and TGI (Text Generation Inference) are the two primary open-source server implementations for high-throughput LLM serving. vLLM generally offers better performance due to its page attention implementation. However, vLLM requires more manual intervention for unsupported models, while TGI may provide easier model swapping. The choice between them depends on specific use cases, model compatibility requirements, and performance needs.

Click to watch →

Deploy FULLY PRIVATE & FAST LLM Chatbots! (Local + Production)

Deploy FULLY PRIVATE & FAST LLM Chatbots! (Local + Production)

37.7K

Text Generation Inference (TGI) from Hugging Face enables deploying large language models locally. Two installation approaches exist: manual installation requiring Rust, Protobuf, Flash Attention, and vLLM (time-consuming), or Docker deployment using a single command. The Docker command maps ports, mounts data volumes for persistent storage, and specifies model IDs. Quantization techniques reduce memory usage to 10GB, enabling deployment on consumer GPUs. This allows running sophisticated models like Falcon 7B on home machines.

Click to watch →

12 Open Source AI Tools That Feel ILLEGAL To Know About

12 Open Source AI Tools That Feel ILLEGAL To Know About

4.6K

vLLM is an industry-standard serving engine for running large language models at scale. Its key innovation is paged attention, which manages model memory like an operating system manages RAM, minimizing wasted memory. This achieves up to 24 times the throughput of naive setups on the same hardware. Continuous batching slots new requests between others mid-flight, keeping GPUs fully utilized. It serves an OpenAI-compatible API, automatically batches incoming requests, and powers a significant portion of self-hosted AI deployments. This technology enables companies to serve thousands of users simultaneously without melting their GPUs.

Click to watch →

Unsloth Setup

Dataset Prep

LoRA Setup

Format Data

Train Model

Training Eval

Export GGUF

Ollama Run

Local Test

Unsloth Setup

0:00

Playing Section

1
Identify 1B and 3B models as suitable for on-device runs.
2
Install the nightly version of Unsloth for fine-tuning.
3
Load the 3B instruct model with 4-bit quantization.

101.5K views1.9K17:35@engineerpromptOriginal Release: 2024-09-29

This tutorial demonstrates how to fine-tune Meta's Llama 3.2 instruct model using the Unsloth library, covering key steps including dataset preparation (using the FineTome-100K dataset), configuring LoRA adapters with appropriate rank and alpha values, adjusting prompt templates to match the model's expected format, and training with the TRL library; the process concludes by converting the fine-tuned model to GGUF format for local deployment using Ollama, enabling users to run custom fine-tuned models on their own devices without cloud dependencies.

[00:00:00]last week meta released Lama 3.2 which is a new family of four different models including multimodal models and they're pretty impressive both for language and vision tasks for their respective sizes but you know what's better than that it's your custom fine tune llama 3.2 that's exactly what we're going to learn in this video we will use unslot for fine tuning then I'll show you how you can run that fine tune model locally using ola because what's the point of a fine tune model if you can't run it locally but before then let's have a quick look at the release block post this new release has two sets of models one are lightweight which is one and 3 billion model and the other set is multimodel with 11 and 90 billion there is no 405b this time meta is moving away from the standard seven or 8 billion models now they have a 11 and 90 billion instead of 8 or 70 billion model but I think the most interesting one are the 1 and 3 billion models because you can run them on device we will look at the 11 and 90 billion models for vision tasks in another video apart from these models meta has also released llama stack which is their opinionated version of how developer experience should look it's great to see that these model providers are now building text TXS for deployment let's talk about how you can fine tune uh one of these smaller models on your own data set and then I'll show you how you can run this locally using o Lama fine tune Lama 3.2 we will use the official notebook from the unslot team I have covered variations of this notebook in my earlier videos for fine-tuning other variance of Lama this is going to be a quick recap of those notebooks first we need a data set uh to fine-tune the model on for this example we're using the fine Tom data set which has 100,000 examples so it's a relatively huge data set and it has multi- turn conversations this data set is collected from multiple different sources so I think it's a very good candidate if you are fine-tuning uh LM in general but if you're fine-tuning this model for your own specific task you will just need to provide your own data set and I'll later on show you how you can structure your data set first we need to install unslot they recommend to use the nightly version which is basically the the latest version unot uses a fast language model class for dealing with llms we're going to load the Lama 3.23 billion instruct model we're using this model because it's a relatively smaller model that you can potentially run on uh on device device such as on a smartphone uh another thing to highlight is I'm using the unslot version you can also use the Llama version directly you'll need to provide your hugging face um token ID and accept their terms and conditions the 11 billion and 90 billion models are not available in all regions and that has to do with its uh Vision capabilities so you need just need to be careful Uno currently does not support Vision models yet but hopefully they will um add support soon when you're loading the model you need to Define three different parameters the first one is the max sequence length in our case we are setting it to 2048 this number is dependent on your training data set look at your training examples and see the maximum sequence length available in your data set and I'll recommend to set it to that but setting it to a higher value will also need more TPU vram so you need to be careful of that data types you can set to floating Point 16 or 8 but if you keep it none it will automatically select depending on your Hardware we're going to be using the 4bit quantization to reduce the U memory usage or memory footprint so here we're loading both the model as well as the tokenizer next I'm adding Lura adopters we are not using full fine tuning even though the model is pretty small we're adding Laura adopters these are different modules that we are targeting we train completely separate modules and then merge it with the original model weights there are a couple of other things to keep in mind one is the r or rank this determines how uh parameters are going to be in your Lowa adapter if you uh set it to a high number this will give you much better uh fine tuning or the performance is usually going to be better but again your um fine tuning a large number of uh in your low adopter so that will mean that you will need more uh resources in terms of vram to fine-tune uh or train the low adopters so usually 16 or 32 provides you um good compromise between the memory footprint and the performance another thing is the impact of this Lura when you merging it back to the original weights of the model so that is set through the Laura Alpha now some points on the prompt template so here's the PR template that the Lama 3.1 and 3.2 uses you need to make sure that your data set that you're providing in order to find T the model actually follows this specific prompt template because we're using the instruct version of the models for fine tuning if you're fine-tuning the base model you can provide your own template but if you're working with instructor chat version then you have to follow the template used by the model itself the promt template expects role and content but here you can see that the data set we're using actually uses another format which is from human and then I think there is from GP team right so it uses a different promt templat so we need to adjust this prompt template and for that you can use the get chat template class or function from unslot basically we provide the token use the prompt template from Lama 3.1 which is uh similar to 3.2 and that will take all the data sets and convert it to our specific prompt template so here we're loading the data set now we need to go from this which is from system and then you provide the value or from human or from GPT to the role based approach everything should be converted to RO system Ro user androll assistant we do that through the standardized share GPT uh function that we just created now if you look at um some example conversations here you can see that we went to the content so here's the content then here's the role role is user and that's the uh question asked by the user then we have a role of assistant and this is the response generated by the assistant when you're are formatting your own data set you will have to follow this specific prompt template in order to fine-tune a Lama 3.1 instruct version another thing is that the Lama 3.1 instruct defaults chat template adds this specific sentence in the system instruction so it's actually telling the model that this cut off training date was in December 2023 and it adds today's date to be uh July 26 so if you see something like this in responses from the model don't be alarmed because that's just part of the system instruction and later on they actually masked this for now in order to train the model we are using the TRL library from hugging face and we're going to be using the supervised fine tuning trainer because we are doing supervised fine tuning in this case so we provide the model name the tokenizer these are coming from the unslot then we provide our data set we also tell it which column to use as basically our prompt template that we have already formatted we added a text column to the data and the maximum sequence length in the training data set now here are some other uh specific parameters a couple of things which I want to highlight is if you set the number of epoch so for example if it sets to one it will go through the whole data set at once only during training but 100,000 examples are a pretty huge data set so that's going to take a while that's why we set the max steps to 60 you can either set the max steps or uh you can set the the number of epoch now what's the relationship between the two that is determined by our batch size in order to get the total number of steps in an Epoch you can divide the size of the data set by the batch size for example if you have 100 examples if you divide it by two you will get a maximum of 50 steps in the EPO we're just running it for 60 St step steps which is the fraction of the total number of steps possible for 100,000 examp examples the reason we do it is because we don't want to run it for a long time I just want to show you an example and that's why you probably are not going to see a greatly trained model in order to get really good training output you definitely want to run it for a lot longer the learning rate determines the speed of convergence if you set it to a high number the training speed is going to be faster but the training might not converge you usually want to find a suite spot where the learning rate is small enough that it converges but that will also take much longer to train okay one more thing that you want to train the model on the output not on the inputs so that's why you want to calculate the loss of the model on the output from the assistant not based on the inputs from the user so the model should see the user input generate a response and then compare the output with the original or gold standard output or ground truth and that's where you compute the loss so this section takes care of that it forces the model to only use the output for uh computation of the training loss or the test loss depending on if you have a test data set now you can look at uh how the tokenized version of the data set looks like you can see that we have clearly added the system role here is the well formatted user input and then we have the well formatted assistant response and this is the data set that we will use to train our mod want to get rid of this part which is the um system message part you can mask that here we're masking that and now you can see that you don't really see the original system message you only see the output the model is supposed to generate okay next we call the train function on the trainer that we created you can see that the loss goes down then comes up again the reason is that we're running it for a way small number of steps probably we can play around with the learning rate as well that will control the speed of convergence these are different parameters that you need to play around with if you're using bigger batch sizes you can set the learning rate to a relatively higher value but bigger batch sizes will also depends on the available GPU vram that you have so there has to be a compromise between these hyperparameters that you're working with okay so after this training you can see that if we run uh this specific um prompt on the train model uh then here is the response that we get so here we see the system message but we'll have to mask that ourselves in terms of the user input here's the user input continue the Fibonacci sequence so we provide the Fibonacci sequence and then the response generated by the model is here now you can also stream this if you want here's an output of the stream response which basically does the same same thing but in a streaming fashion okay once you train the model you can either push this to GitHub or store it locally I'm mostly interested in how to store the GG verion of the model because I want to load this in olama and run it locally for that to work you just need to call the save pre-trained ggf provide the model name I'm calling it model uh 3 billion provide the token and the level of quantization since it's a atively smaller model I wanted to run it in uh 16bit floating Point Precision keep in mind this step will take quite a long time because it has to First download and install Lama CPP and then convert uh this model to GG UF format so here's the model that I downloaded from Google collab if you run the training locally you are going to see unslot fp16 ggf I downloaded the model from Google collab now let me show you the rest of the process next let me show you how to run that trained file locally using ama ama uses the concept of model file which is basically a set of configurations that you will need to provide for AMA to use a model locally there are a number of different things you can use uh there's an instruction called from where you tell it which model to use you can set different uh parameters such as temperature uh Max context window and so on you can also provide the full prompt template for the uh model here is a quick example if you want to use uh L 3.2 with different configurations than the default you're going to say from Lama 3.2 here they're changing the temperature to one the contact Max contact window is changed to 4096 and you can also provide a simple system instruction if you go to any model on AMA you can see this template if you click on this this is the model file used by any model available on AMA I have downloaded the ggf file that was created after uh fine-tuning the model on um Google collab notebook you just want to look at the file that is uh GF so downloaded here and then created another file called Fine Lama in here I'm saying from and then providing that model name with uh GG at the end so this is basically the model file that we're going to be using we can also include a temp template that will Define The Prompt template but since it's already in the tokenizer so I don't need to do that you can also Define the system prompt but in this case we want to mask it so I'm not going to add that either now you need to have Ama up and running after that you need to provide some details to create this model in ama we type the command AMA create then what you want the model to be called so I'm going to call it then you can use this DF parameter and you need to provide the path of the uh model file that we created so it's in the same directory when you click on it this will uh start transferring the data and if everything goes well uh it will create a model file for us it's using the template from Lama 3 instruct seems like everything is successful now we can run our model but before that let me show you if this shows up in the model list so here we have have the fine Lama this is basically the model that we just created I have a whole bunch of other models that I have already downloaded and now in order to run this model all we need to do is just type AMA run and just like any other AMA model we just need to provide the name now it's a 3 billion model so it's going to be extremely fast if I say hi you can say uh that it generates responses pretty great and pretty quickly all right so I'm going to ask it to write a program in Python to move files from uh three to a local directory and you can see it's really fast because it's just a 3 billion model that is running completely locally and that's the finetune model that we just fine tuned okay so this was a quick video on how to fine tune Lama 3.2 using Onslaught and then run it locally on your own machine using AMA I hope uh this was helpful I'll put a link to the um Google collab in the video description in this video I only focused on the 3 billion uh same approach will apply app to the 1 billion model for uh 11 and 90 billion models the approach is a little different because it has a adopter for the uh Vision component so the same approach probably is not going to apply but I'm going to be creating some videos specifically focused on the vision model because I think there are some great applications there specifically for vision based rag which is a topic I'm personally interested in if that interests you make sure to subscribe to the Channel I hope you found this video useful thanks for watching and as always see you in the next one

Up Next

Top Tech Skills for 2025-26: Fullstack, DevOps, AI Guide

@harkirat1

125.4K views•2025-12-08

BitTorrent Protocol Explained: Piece Selection & Peer Choking

@StevenGordonAU

481 views•2013-02-22

HTTP Requests Explained: GET, POST, PUT, DELETE

@codecademy

103.1K views•2021-10-07

Enigma Machine Mechanics: WWII Encryption Explained

@JaredOwen

13.2M views•2021-12-11

Related Study Plans & Knowledge Roadmaps

Structured learning paths in Computer Science

View All Computer Science Topics

computer-science

Building BitTorrent: Go, P2P & Concurrency

01Go Fundamentals & Concurrency

02Networking Foundations & P2P Architecture

03Bencode Parsing & Torrent Metadata

+ 3 more learning modules

Explore Roadmap

computer-science

Actor Models (C++ Concurrency & Lock-Free)

01C++ Multi-Threading Foundations

02Thread Pools and Task Scheduling

03Atomics and Lock-Free Queue Design

+ 3 more learning modules

Explore Roadmap

computer-science

Coding HM Type Inference: Algorithm W & OCaml

01Introduction to Functional Programming & OCaml

02Abstract Syntax Trees & Language Interpreters

03Type Systems & Lambda Calculus

+ 2 more learning modules

Explore Roadmap

computer-science

Building OAuth 2.0: Node.js OIDC Server

01Web & Security Foundations

02Authentication vs. Authorization & JWTs

03The OAuth 2.0 Protocol & Grant Types

+ 3 more learning modules

Explore Roadmap