Academic Databases
Explore Our Semantic Databases
STEM & Tech
Math, Physics, CS & Engineering...
Economics
Macro, Micro, Econometrics...
Business
Management, Marketing, Finance...
Health & Life Sciences
Biology, Medicine, Neuroscience...
Humanities & Social Sciences
Psychology, Law, Philosophy, History...
General Studies
General & interdisciplinary studies...
Explore Topics
Features

Build a Local RAG Chatbot with Arcee Trinity Mini and LangChain

Added: 2026-08-06

Deep Dive

Fundamental concepts of Retrieval-Augmented Generation (RAG), specifically how a standard ingest-retrieve-generate pipeline functions.

RAG Fundamentals and Advanced Techniques – Full Course

RAG Fundamentals and Advanced Techniques – Full Course

175.5K

Retrieval-Augmented Generation (RAG) is a framework combining retrieval-based and generation-based models to produce more accurate, contextually relevant responses. The core concept involves injecting custom data into large language models so they can answer questions about specific contextual information beyond their training data. RAG consists of two main components: the retriever identifies and retrieves relevant documents, while the generator combines retrieved documents with queries to generate responses. The workflow involves parsing documents, chunking them, creating embeddings via embedding models, storing embeddings in vector databases, transforming queries into embeddings, searching for similar items, and generating final answers augmented by retrieved information.

Click to watch →

Introduction To Undertsanding RAG(Retrieval-Augmented Generation)

Introduction To Undertsanding RAG(Retrieval-Augmented Generation)

162.7K

Retrieval-Augmented Generation (RAG) is a technique that optimizes large language model outputs by referencing an authoritative external knowledge base before generating responses, addressing two key limitations of standalone LLMs: (1) hallucination when answering questions about events outside the model's training data timeframe, and (2) inability to access proprietary or organizational data without expensive fine-tuning. The RAG pipeline consists of two main components: a data injection pipeline that processes external data (PDF, HTML, Excel, SQL, etc.) through parsing, chunking, and embedding into a vector store, and a retrieval pipeline that converts user queries into vectors, performs similarity search in the vector database to retrieve relevant context, and augments the LLM prompt with this context before generating the final output.

Click to watch →

RAG Crash Course with LangChain: A 2.5 Hour Practical Guide

RAG Crash Course with LangChain: A 2.5 Hour Practical Guide

Retrieval-Augmented Generation (RAG) is a technique that optimizes large language model outputs by referencing an authoritative external knowledge base outside the model's training data, thereby preventing hallucinations and enabling integration of organizational data without expensive model retraining; the complete RAG pipeline consists of two main stages: the data injection pipeline (which involves document parsing, chunking, and embedding into a vector database) and the retrieval pipeline (which converts user queries to vectors, retrieves relevant context from the vector store, and augments LLM prompts with this context before generation).

Click to watch →

Don't learn AI Agents without Learning these Fundamentals

Don't learn AI Agents without Learning these Fundamentals

405.3K

RAG combines retrieval, augmentation, and generation to enhance AI responses with up-to-date information. The process works as follows: (1) Retrieval - convert the user's question into an embedding and perform semantic search against document embeddings to find relevant documents; (2) Augmentation - inject retrieved documents into the prompt at runtime, providing the AI with real, up-to-date, private data; (3) Generation - the AI generates a response using both the original question and the retrieved documents. This approach allows AI assistants to rely on current company data without requiring model fine-tuning.

Click to watch →

RAG and MCP Fundamentals: A Practical AI Systems Course

RAG and MCP Fundamentals: A Practical AI Systems Course

Retrieval-Augmented Generation (RAG) is an AI system architecture that combines retrieval from external knowledge sources with large language model generation to produce accurate, context-aware responses. Unlike traditional prompt engineering or fine-tuning, RAG dynamically retrieves the most relevant information at query time using semantic search and vector databases, making it ideal for handling dynamic factual information where policies frequently change. The RAG pipeline involves three core stages: (1) document retrieval using semantic search with embedding models, (2) context augmentation by combining retrieved information with user queries, and (3) response generation by the LLM. Effective RAG implementation requires proper document chunking strategies, vector database indexing (such as HNSW), and caching mechanisms to optimize performance in production environments.

Click to watch →

Deep Dive

The theory of vector embeddings, semantic search, and the mechanics of querying vector databases.

OpenAI Embeddings and Vector Databases Crash Course

OpenAI Embeddings and Vector Databases Crash Course

680.1K

Searching a vector database involves three steps: (1) Identify what you want to search for, (2) Create an embedding for your search term, (3) Perform a search against existing embeddings. The SQL query selects text and vector columns, converts the vector to a score using JSON_ARRAY_PACK, orders by score in descending order, and limits results. The scores represent how similar the search term is to existing vectors in the database, with higher scores indicating closer similarity.

Click to watch →

DEF CON 33 - Exploiting Shadow Data from AI Models and Embeddings - Patrick Walsh

DEF CON 33 - Exploiting Shadow Data from AI Models and Embeddings - Patrick Walsh

115.5K

Vector embeddings are mathematical representations created by embedding models that convert text into fixed-size arrays of numbers (typically 300-3,000 dimensions). Documents are chunked into sentences or paragraphs, and each chunk generates an embedding vector. Similar concepts appear close together in vector space (e.g., 'I'm happy today' and 'I feel great' are near each other), while dissimilar concepts appear far apart. Vector databases use these embeddings to perform semantic searches - when a user asks 'What's the irrigation status?', the system finds the closest matching vectors (like 'I need to fix the sprinklers') and retrieves relevant context. This technology powers facial recognition, image tagging, multimodal search, and document classification across virtually all modern software systems.

Click to watch →

ChatGPT Prompting vs RAG vs fine tuning: Aprovecha la IA al máximo | #laFunción 9x18

ChatGPT Prompting vs RAG vs fine tuning: Aprovecha la IA al máximo | #laFunción 9x18

58.1K

This segment covers the technical foundations of embeddings and vector databases: (1) Embeddings as numerical representations capturing semantic meaning of words and phrases, (2) Vector databases for storing and querying embeddings efficiently, (3) Chunking strategies - fixed-size chunks with overlap, semantic units (chapters, clauses), and structure-based chunking, (4) The relationship between embeddings and RAG - how semantic vectors enable finding relevant information. The segment explains how these technologies work together to enable semantic search and retrieval in AI systems.

Click to watch →

Let's Build a Local RAG System with Ollama & Qdrant

Let's Build a Local RAG System with Ollama & Qdrant

13.2K

Vector embeddings convert text into numerical representations that capture semantic relationships in multi-dimensional space. The RAG workflow involves four components: a large language model, a vector database, an embedding generation process, and the source data. When a user asks a question, the system converts the prompt into a vector, queries the database for similar stored vectors, retrieves associated data, injects this context into the prompt, and generates a response. This enables semantic search where meaning—not exact keyword matches—is what determines relevance.

Click to watch →

What is a Vector Database? Powering Semantic Search & AI Applications

What is a Vector Database? Powering Semantic Search & AI Applications

667.9K

Vector databases solve the semantic gap by representing data as mathematical vector embeddings, which are essentially arrays of numbers. These vectors capture the semantic essence of data where similar items are positioned close together in vector space and dissimilar items are positioned far apart. With vector databases, similarity searches can be performed as mathematical operations, looking for vector embeddings that are close to each other, which translates to finding semantically similar content.

Click to watch →

Deep Dive

Basic proficiency with the LangChain framework, particularly its abstractions for document loaders, text splitters, and vector stores.

The LangChain Cookbook - Beginner Guide To 7 Essential Concepts

The LangChain Cookbook - Beginner Guide To 7 Essential Concepts

376.4K

Document processing in LangChain involves loading external data (via loaders like Hacker News), splitting long documents into chunks (using recursive character splitters with configurable size and overlap), and storing embeddings in vector stores. Vector stores (like Pinecone, Weaviate) enable efficient similarity search by storing numerical representations of text chunks. This pipeline transforms raw documents into searchable, model-ready data, enabling applications like document retrieval and RAG systems.

Click to watch →

Building the Future with LLMs, LangChain, & Pinecone

Building the Future with LLMs, LangChain, & Pinecone

75.1K

LangChain provides abstractions for document processing including document loaders that ingest text from various sources (files, web pages, Notion) into a common format. This format includes the text content and metadata (source, page, relative position). The abstractions make it easy to swap out different components like embedding models or vector databases.

Click to watch →

AI AGENTS AND FRAMEWORKS

AI AGENTS AND FRAMEWORKS

160

LangChain provides various loaders for different document formats: PDF loader, Wikipedia loader, text file loader, and web-based loaders. For splitting documents, LangChain offers multiple splitters including recursive character splitter (most commonly used), character text splitter, recursive JSON splitter, and HTML text splitter. The recursive character splitter is widely used because it captures each word accurately during the splitting process. Embedding models include open-source options (Hugging Face models) and paid models (Amazon Titan embeddings, Azure OpenAI embeddings). Vector databases include open-source options (Chroma, FAISS) and paid options (Pinecone, Azure, Amazon).

Click to watch →

LangChain Full Course in One Shot | Learn RAG, LangChain, and AI App Development (Complete Tutorial)

LangChain Full Course in One Shot | Learn RAG, LangChain, and AI App Development (Complete Tutorial)

13.6K

Document loaders ingest data from external sources into LangChain documents with page content and metadata. Categories include: (1) File-based loaders: TextLoader, PyPDFLoader, CSVLoader, JSONLoader, UnstructuredFileLoader; (2) Web-based loaders: WebBaseLoader, SitemapLoader; (3) Database loaders: DataFrameLoader, SQLLoader. Text splitters break large documents into chunks for efficient LLM processing. Types include: (1) Character splitter; (2) Recursive character splitter (most common); (3) Markdown header splitter; (4) HTML splitter. Embeddings are numerical text representations as vectors in high-dimensional space (384-3072 dimensions), capturing semantic meaning. Vector stores index and search information based on vector representations, enabling semantic search. Retrievers accept natural language queries and return relevant documents, more general than vector stores. Types include: (1) Vector retrievers using embedding + vector store; (2) Sparse retrievers (BM25) using keyword matching; (3) Ensemble/hybrid retrievers combining multiple strategies; (4) Contextual compressors filtering by similarity threshold; (5) Multi-query retrievers generating LLM reformulations for comprehensive results. Building complete RAG applications involves: modular project structure, helper functions for loading/chunking/embedding, vector store and retriever implementation, prompt templates with system/user prompts, chains combining retrievers with LLMs, and deployment with Flask backend and HTML/CSS frontend.

Click to watch →

Introduction to Langchain?

Introduction to Langchain?

466

LangChain is a framework that simplifies working with Large Language Models by providing an abstraction layer over different LLM providers' APIs. Different providers have different API specifications, but LangChain allows developers to simply specify the provider name and model name, then use standardized classes. The framework provides built-in components including document loaders for various file formats, text splitters, vector database integration (Chroma), prompt templates, and output parsers. This abstraction eliminates the need to understand each provider's unique API structure, making LLM application development much simpler and more maintainable.

Click to watch →

Deep Dive

Familiarity with local LLM execution environments (e.g., Ollama, Llama.cpp) and setting up python-based environments on macOS.

LLM Chat App in Python w/ Ollama-py and Streamlit

LLM Chat App in Python w/ Ollama-py and Streamlit

11.6K

Ollama recently announced Python and JavaScript libraries that make LLMs more accessible with just a couple lines of code. The Python library wraps the Ollama API and provides types for response data, giving access to the entire Python ecosystem. While Bash is nice for CLI interfaces, it's not how most people want to interact with LLMs. Even UIs like Open Web UI don't provide programmatic access to the underlying API. To set up, confirm Python version is over 3.8, create a virtual environment using 'python -m venv .venv', activate it with 'source .venv/bin/activate', and install Ollama with 'pip install ollama'.

Click to watch →

Ollama Installation + Run LLM in Python | Local AI Setup for Beginners | Hindi | Vivek Manglani

Ollama Installation + Run LLM in Python | Local AI Setup for Beginners | Hindi | Vivek Manglani

154

Ollama is an open-source platform that allows users to run large language models locally on their computers without relying on cloud services, offering complete privacy and free access to models like Llama 3.2, Mistral, and Gemma; the platform supports various model sizes from 1 billion to 45 billion parameters, with performance depending on available VRAM and CPU/GPU resources, and can be integrated with Python for custom applications.

Click to watch →

How To Build an API with Python (LLM Integration, FastAPI, Ollama & More)

How To Build an API with Python (LLM Integration, FastAPI, Ollama & More)

125.8K

Ollama is a free, open-source tool that allows you to run large language models locally on your own computer. To set it up, first download and install Ollama from its official website. Once installed, verify it works by typing 'ollama' in the terminal or command prompt. To use a specific model, pull it using 'ollama pull [model_name]' (e.g., 'ollama pull mistral'). Then run the model with 'ollama run [model_name]'. This enables you to test and use LLMs locally without relying on cloud providers, giving you full control over the environment and reducing costs.

Click to watch →

Ollama Tutorial - Run LLM Locally in MacOS Linux Windows

Ollama Tutorial - Run LLM Locally in MacOS Linux Windows

5.1K

Ollama is an open-source tool that enables users to run large language models (LLMs) locally on Linux, Mac OS, and soon Windows systems, supporting models like Llama 2, Code Llama, and others with quantized versions requiring only 4GB of storage; the tool provides a simple command-line interface for model installation, interaction, and management, with GPU acceleration available for faster inference.

Click to watch →

اجرای لوکال LLM ها در OLLAMA و پایتون (بدون نیاز به API خارجی!) 🚀

اجرای لوکال LLM ها در OLLAMA و پایتون (بدون نیاز به API خارجی!) 🚀

355

Ollama is an open-source platform that enables local execution of powerful language models like Llama, Mistral, and Gemma without requiring external API connections or data transmission. Users can install Ollama, select and download models from the command line, and interact with them through a terminal interface. For Python integration, developers can create a virtual environment, install the Ollama Python package, and write code to connect to local models, send prompts, and receive responses. Advanced features include streaming responses for real-time output display, enabling the development of applications like chatbots, text summarizers, and content generation tools entirely on the local system.

Click to watch →

Local RAG Setup

App and Data

Vanilla Chat Test

Similarity Search

MMR Diversity

Hybrid Retrieval

Query Rewriting

Reranking Chunks

Combined Techniques

Code and Files

Local RAG Setup

0:00

Playing Section

1
Explains benefits of local models: privacy, no internet, no API costs.
2
Introduces RC Trinity Mini 26B model with MoE architecture.
3
Uses 8-bit quantized version via Llama CPP for the demo.

Deep Dive

Evaluating RAG pipeline performance using specialized frameworks like Ragas or TruLens to measure context precision and faithfulness.

Developing Applications with LangChain | Master LLMs, RAG, and Agentic Workflows

Developing Applications with LangChain | Master LLMs, RAG, and Agentic Workflows

4.2K

RAG evaluation uses metrics from the Ragas framework: faithfulness measures whether generated outputs accurately represent retrieved documents, calculated as faithful claims divided by total claims; context precision measures how relevant retrieved documents are to the query. Both metrics normalize between 0 and 1, with higher values indicating better performance. Evaluation helps identify whether retrieval fails to find relevant documents or if generation produces hallucinations, guiding improvements to either the retrieval or generation component.

Click to watch →

A Survey of Techniques for Maximizing LLM Performance

A Survey of Techniques for Maximizing LLM Performance

220K

The RAGAS framework evaluates RAG systems across four metrics: (1) Faithfulness - checks if answer facts can be reconciled with retrieved content, identifying hallucinations; (2) Answer Relevancy - measures whether the model uses retrieved content appropriately relative to the original question; (3) Context Precision - evaluates the signal-to-noise ratio of retrieved content by determining which content blocks were actually used in the answer; (4) Context Recall - measures whether the relevant information needed to answer the question exists in the content. These metrics help identify whether to optimize retrieval, add reranking, or adjust embeddings.

Click to watch →

RAG using Milvus, HuggingFace, LangChain, Ragas, with or without OpenAI

RAG using Milvus, HuggingFace, LangChain, Ragas, with or without OpenAI

905

Ragas is an open-source RAG evaluation framework that uses LLMs as judges to evaluate RAG systems. It measures multiple metrics: (1) Faithfulness - how well the answer corresponds to the retrieved context, (2) Context Precision - how many retrieved chunks are relevant, (3) Context Recall - how many relevant chunks were retrieved, and (4) Answer Relevance - how well the answer addresses the query. The framework can evaluate both the overall system and individual components, requiring only about 20 examples instead of thousands of human-labeled examples.

Click to watch →

(Renewal) RAG 기초부터 advanced ( BM25 & 평가 포함) 세미나 (2분 18초부터~~ )

(Renewal) RAG 기초부터 advanced ( BM25 & 평가 포함) 세미나 (2분 18초부터~~ )

1.1K

This section covers RAG evaluation: (1) RAG evaluation requires assessing both retrieval quality and response quality; (2) Key metrics include retrieval metrics (precision@K, recall@K, MRR), response metrics (BLEU, ROUGE), and human evaluation; (3) Golden data refers to ground truth answers for evaluation; (4) Popular frameworks include RAGAS, TruLens, and DeepEval; (5) DeepEval provides metrics for faithfulness, answer relevance, context precision, and context recall; (6) Automated evaluation has limitations and should be complemented with human evaluation.

Click to watch →

How Production RAG Systems Are Monitored

How Production RAG Systems Are Monitored

356

Production RAG systems require automated evaluation frameworks like RAGAS and TruLens to assess faithfulness, hallucination, context precision, and observability, as manual checking of thousands of responses is impractical; RAGAS excels at generating synthetic test data from documents while TruLens provides broader pipeline observability including latency tracking and feedback monitoring, though both frameworks rely on LLM-based evaluation approaches that depend on quality test datasets and careful metric design.

Click to watch →

Deep Dive

Implementing Agentic RAG, which introduces reasoning loops, tool-calling, and autonomous decision-making to the retrieval flow.

Building Agentic RAG From Scratch in Pure Python

Building Agentic RAG From Scratch in Pure Python

2.9K

Agentic RAG is an approach that replaces or extends semantic RAG for scenarios requiring company data, private data, or any information to be made available to large language models during AI automation. Unlike semantic RAG which is better for low-latency and cost-optimized scenarios, agentic RAG outperforms when time and budget allow due to its feedback loop mechanism. The simplest implementation uses markdown files on the file system combined with three essential tools: a tool to list files, a tool to search files, and a tool to read files. The path library provides methods to handle file paths safely, with parent() moving up one directory level and resolve() ensuring clean paths. The glob() function finds files matching a pattern in a directory, while relative_to() converts full paths to relative paths to save tokens. The grab() function uses regular expressions to search for patterns within files, with re.compile() creating reusable patterns and re.IGNORECASE enabling case-insensitive matching. The read_file() function processes files line-by-line using splitlines() and enumerate() for human-readable line numbers, with results stored as tuples containing filename, line number, and line content. Before reading files, the function validates that the target path is within the designated notes directory using is_relative_to(), ensuring the agent remains contained within a specific folder.

Click to watch →

Agentic RAG Tutorial: Building AI Agents with Function Calling and Retrieval-Augmented Generation

Agentic RAG Tutorial: Building AI Agents with Function Calling and Retrieval-Augmented Generation

8.4K

Agentic RAG (Retrieval-Augmented Generation) enhances traditional RAG systems by enabling autonomous agents to make decisions about when to use external knowledge bases versus their own reasoning, allowing for multi-step search processes and function calling to access tools; this approach transforms static retrieval pipelines into dynamic, reasoning-capable agents that can perform iterative research, reformulate queries, and adaptively choose between using pre-existing knowledge or generating new insights.

Click to watch →

5-Getting Started With Agentic RAG With Detailed Implementation Using LangGraph

5-Getting Started With Agentic RAG With Detailed Implementation Using LangGraph

23.7K

Agentic RAG (Retrieval-Augmented Generation) is an advanced AI architecture where autonomous agents dynamically decide whether to retrieve external data or generate answers directly, unlike traditional RAG's fixed retrieve-then-generate pipeline; this implementation uses LangGraph to create a workflow where an agent node makes intelligent decisions about retrieval based on query analysis, then routes to either a retriever node or a generator node accordingly.

Click to watch →

Full Course (Lessons 1-10) AI Agents for Beginners

Full Course (Lessons 1-10) AI Agents for Beginners

368.1K

Agentic RAG extends basic retrieval-augmented generation by enabling agents to analyze queries, create task plans, and iteratively retrieve information until sufficient for answering. Key features include: multi-step problem solving for complex questions, verification loops to ensure adequate information, and long-term memory to avoid redundant steps. Implementation combines Azure Search for document retrieval with custom plugins for dynamic data. The agent first reviews retrieved context, then calls available functions if needed, enabling responses that combine both retrieved documents and real-time data.

Click to watch →

Agentic RAG - ПОЛНОЕ ОБЪЯСНЕНИЕ! Где он лучше, чем обычный RAG?!

Agentic RAG - ПОЛНОЕ ОБЪЯСНЕНИЕ! Где он лучше, чем обычный RAG?!

4.3K

Agentic RAG systems consist of a system prompt, user query, tools, and multiple data sources. Tools include Vector Search (transforming queries to embeddings and performing similarity search), Knowledge Graph Search (querying entity relationships), and Calculator (performing numerical calculations). Data sources include vector databases (book descriptions, reviews, author information) and knowledge graphs (relationships between authors, categories, and purchase patterns). The agent acts as an intermediary coordinating between user queries, tools, and data sources to produce comprehensive responses.

Click to watch →

Deep Dive

Optimizing and scaling local LLM deployment for production-level throughput using high-performance inference engines like vLLM.

Optimize LLM inference with vLLM

Optimize LLM inference with vLLM

10.2K

vLLM is an open-source inference engine that optimizes large language model serving through three core innovations—PagedAttention for efficient memory management, Continuous Batching for asynchronous request processing, and Prefix Caching for reusing common query prefixes—which together deliver up to 24x higher throughput compared to standard transformer implementations, enabling faster, more scalable, and cost-effective deployment of generative AI applications.

Click to watch →

Scalable and Efficient LLM Serving With the VLLM Production Stack - Junchen Jiang & Yue Zhu

Scalable and Efficient LLM Serving With the VLLM Production Stack - Junchen Jiang & Yue Zhu

317

The VLLM Production Stack addresses the four major challenges of large language model inference—high cost, slow performance, difficulty adapting to new models, and painful self-hosting—through a three-layer architecture: a production stack for distributing VRM serving engines, a VRM serving engine supporting latest models and hardware, and an ARM cache layer that optimizes KV cache management through compression, sharing, and efficient storage/retrieval across GPU and CPU memory, combined with a one-click deployment interface for easy self-hosting on any cloud.

Click to watch →

Deploying Local LLM but It Is Slow? Here's How to Fix It (Hopefully) | LLMOps with vLLM

Deploying Local LLM but It Is Slow? Here's How to Fix It (Hopefully) | LLMOps with vLLM

1.3K

The vLLM library significantly improves the inference speed of Large Language Models compared to the standard Hugging Face Transformers library by implementing advanced optimizations such as paged attention and built-in quantization support, enabling faster deployment and more efficient resource utilization for production AI applications.

Click to watch →

Fast Inference, Furious Scaling: Leveraging VLLM With KServe - Rafael Vasquez, IBM

Fast Inference, Furious Scaling: Leveraging VLLM With KServe - Rafael Vasquez, IBM

448

vLLM is an open-source inference engine from UC Berkeley that speeds up LLM output through three key techniques: (1) Paged Attention - addresses the KV cache memory bottleneck (which stores word relationships to avoid recomputation) by breaking the cache into flexible chunks like OS virtual memory paging, reducing memory waste from 60-80% to just 4%; (2) Continuous Batching - processes requests dynamically as they arrive rather than waiting for complete batches, keeping GPU resources constantly working; (3) Optimized CUDA Kernels - custom code specifically optimized for KV cache operations, improving GPU resource efficiency. These optimizations make vLLM significantly faster and more cost-effective for LLM inference compared to traditional approaches.

Click to watch →

Optimize for performance with vLLM

Optimize for performance with vLLM

2.5K

vLLM is an open-source LLM inference engine developed by Red Hat that optimizes model serving through activation quantization (quantizing both weights and activations), enabling efficient deployment across diverse hardware accelerators including Nvidia, AMD, Intel, Google TPUs, and AWS Trainium, thereby improving both memory utilization and token throughput for large-scale AI workloads.

Click to watch →

Local RAG Setup

App and Data

Vanilla Chat Test

Similarity Search

MMR Diversity

Hybrid Retrieval

Query Rewriting

Reranking Chunks

Combined Techniques

Code and Files

Local RAG Setup

0:00

Playing Section

1
Explains benefits of local models: privacy, no internet, no API costs.
2
Introduces RC Trinity Mini 26B model with MoE architecture.
3
Uses 8-bit quantized version via Llama CPP for the demo.

19.3K views3225:36@juliensimonfrOriginal Release: 2026-01-09

This video demonstrates building a fully local RAG chatbot using Arcee's Trinity Mini (26B parameter MoE model) running on Apple Silicon with llama.cpp, showcasing advanced retrieval techniques including hybrid search combining vector similarity and BM25 keyword matching, MMR (Maximal Marginal Relevance) for diverse results, query rewriting to improve retrieval accuracy, and cross-encoder reranking for precision, all without cloud APIs or usage costs.

[00:00:00]Hi everybody, it's Julian. A few months ago, I showed you how to build a rag chatbot using RC conductor for intelligent model routing in the cloud.

[00:00:10]But today, we're going fully local. I'm going to keep the same rag architecture with lang chain chroma gradio for the UI, but I'll be using RC Trinity Mini running locally on my Mac with Llama CPP. I've also added some powerful retrieval features since the last video such as hybrid search query writing or reranking. Let's get started. So why would you want to run a local model instead of using cloud APIs? I can think of a few reasons. Uh the first one of course is you don't need internet access. uh you could be on a plane or you could be at customer premises where you don't have internet access and still get work done on your local machine. Uh second of course um you don't need to pay for API calls. No paper token just run on your local machine. And third I guess is privacy. Uh documents are on your on your local machine and you don't need to send them to uh any third party APIs. So there are some good reasons why you would want to work strictly locally.

[00:01:20]So here I'm going to use RC Trinity Mini which I already discussed. It was launched by RC about a month ago now and it's a 26 billion parameter model with a mixture of experts architecture and it was specifically trained on high quality data for enterprise usage. And the cool thing is it's got an Apache 20 license and the model is available on hugging face and uh it's also available in GGUF format um with different quantizations and in fact in my demo I'm going to use the 8bit version of the model. Okay.

[00:02:01]Okay. So, you can just go download that from hugging face, launch it locally with llama CPP like I'm going to do here and uh and you're good to go. You can also try smaller versions. I would say uh smaller quants like six bits or five bit. Um there might be a bit of quality degradation, but honestly, it's it's worth a shot. Maybe it's good enough uh for the job at hand. But here I'm I'm using the uh the 8 bit model which is you know still fairly large but um you know small enough to run on my machine.

[00:02:37]Okay. Um I'm also using Chromo which is a very cool uh vector DB if you want to call it that. And uh and I'm going to show some uh some of the uh more advanced retrieval features available in Chroma. Okay. And as usual I'll use radio for the UI. So, um let's take a quick look at the app first. Um this is what it looks like. Okay. Um it's a simple chatbot. I can enable rag mode or vanilla mode. Okay. I've got obviously an input box here. I've got some sample prompts. Okay. And uh and I'm running this on a collection of energy uh related documents. So, um I actually downloaded uh let me find those docs.

[00:03:26]Here they are. I have about I think it's 94 or 95 PDFs from the um energy u uh domain. I think it's the World Energy Agency.

[00:03:39]Okay. And um it's about 10,000 pages if I recall correctly. Okay. and they're indexed uh in Chromadb and I can go and uh and query those. Okay, so you can use any um you can use any files. I'll I'll I'll explain that later. But that these are the ones I'm using right now. Okay, so my chatbot is running as a Python app um on radio. So default port is 7860 and I've got Lama CPP running here locally port 8080 on that 8bit Trinity Mini uh model. Okay, so these are the building blocks and again I'll put all the links in the video description.

[00:04:22]Okay, so let's uh start testing the app and and looking at the code and figuring out how things work. Okay, let's start in vanilla mode. So no rag, right? just sending the question to the model. So let's try let's try this one. What are the key policy recommendations for accelerating clean energy transitions?

[00:04:44]Okay, so we send that to the model. We see the model doing its thing. My Mac is fairly busy which is fine. And then we're generating an answer. Okay, so typical uh model invocation.

[00:04:59]This is fast. Um it is 58 tokens per second which is great. Um and if we look at the answer there's certainly nothing wrong with it. Um but you know it feels a little high level. Um I'm sure Trinity Mini saw some energy domain data. Um but how do I know this is the you know the best answer? Maybe it's outdated. Maybe it's a imprecise. Unfortunately, maybe it's hallucinating.

[00:05:30]I'm not getting any numbers here. Um but sometimes you get numbers and you go yeah is can I really trust this? And um and that's why you know rag is such a powerful technique. So let's enable rag.

[00:05:46]Okay. And we can pick from different retrieval techniques. So let's start with similarity which is exactly what the name says. We'll try to find content in the knowledge base in the chroma. um uh document collection that is as close as possible to my query. Okay, so let's just go and try this one again.

[00:06:09]So now we're retrieving context as you're seeing here. Okay, and we're passing the context to the model and now it's generating.

[00:06:19]Okay, so there's just a bit of extra latency because of course we need to go and query um the knowledge base. I'm using uh a small embeddings model from from hugging face as you would expect but it's a fairly fast operation right so let's look at the answer now um so we have context right we see the context is quoted directly in the answer so we could go and check page 56 and uh and page 18 of those docs and see for ourselves that these are actually the policies mentioned in the dog. Okay. And uh I'm also printing out the actual retrievalss here. Okay. And uh the highest uh score or the most relevant is the one with that little star. Okay. So that's I think that's useful to see.

[00:07:13]Okay. What did we retrieve? Uh is this really relevant? Is my retrieval mechanism really working? I like to do that. uh more transparency is always better. Okay. So, let's try another retrieval technique.

[00:07:31]Let's clear this.

[00:07:33]Okay.

[00:07:34]And now we're going to try MMR. Okay.

[00:07:37]So, I need to explain what MMR is. So, MMR means maximal marginal relevance.

[00:07:44]So where similarity is looking for the closest match between the the query and the the document chunks, MMR will actually look for diversity. It will fetch a number of candidates and it will select uh chunks that are relevant but diverse. Okay. So for research work, for exploration work, um this is quite useful. um you're not asking you know very pointy question where you know the answer is just you know just a number or stat. Uh here you in this particular case we're trying to get a sense of what those policies are. So maybe exploring and just not strictly looking for similarity match is interesting. So let's give it a shot. Okay. So we're quering the the collection again. Um and uh we're probably going to return different chunks and see uh see what we get here.

[00:08:50]And you can see this is a different answer. We're actually quoting from different documents, right? And um and well, I I guess both answers will be would be interesting. And um and you know, mixing and matching the two would be great. Uh so uh you know why would you stick to uh to one retrieval technique uh when when several are available in chroma and it makes no difference to to trinity you just you're just passing different chunks and and then it's uh you know it's doing its thing right so if you want to relax the similarity constraint uh you can do that and as always uh we have hybrid mode so let's just try hybrid mode clear And as you would expect, hybrid is a mix of the two, right? And there's a there's a parameter to uh to decide, you know, how much MMR you want in there and how much similarity you want in there. Okay.

[00:09:48]Um we'll just uh stick to default. So let's h try this one and see if we get yet another answer to the question.

[00:10:12]Okay. Well, it it I don't know if it looks like a mix. Um, looks like it did hit different documents. We still get that uh US 2024 doc, but this one I think is again a different one. Um, so again, there's no black or white answer to this. Uh I think generally if your rag wants to help users answer very pointy questions like data questions uh you know what's the what's the margin in Q3 for company XYZ okay probably similarity works um if you're doing you know open questions maybe like this one um maybe MMR opens up um the uh the the universe a little bit more and I guess hybrid again is is a mix of the two. So um you know you could you could try all of them um and see what works better for a particular class of questions. Okay. And you you could actually expose that to users, you know, and maybe explain a little bit what those things are and uh and over time learn that well um one retrieval technique works better for a certain group of users and then make that one default. Okay. Um and we could ask follow-up questions, right? Let's let's try follow-up question here. Um something like this. Are there any China um specific policies. Why not?

[00:11:49]Okay, we're we have history management here. So, um you should be able to follow up on that.

[00:11:56]Let's see what we get.

[00:12:02]Okay, so in that knowledge base, we don't have any specific policies for China.

[00:12:09]we just have a few uh a few data points but again they are clearly um pinpointed and and we could go and double check right so this will build a lot of confidence of course in the system um instead of just getting an answer where you know you're never sure how much you can trust the model okay so let's look at a few more techniques to further improve our rack system and now I'm going to show you two pretty cool techniques. The first one is called query rewriting and the other one is called reranking. So let's start with query rewriting.

[00:12:48]That's exactly what the name says. Um before we query the document collection, we are going to rewrite it to improve our um hit ratio I suppose. So why would we do that? Um, you know, user queries have a lot of useless words. Um, and and we don't want to match those. Um, we want to match the key topics and maybe if it's a short query, we want to expand a little bit on it to uh try and find, you know, more chunks, more relevant chunks. So, instead of teaching your users to write good queries, um, let's use the model. So, in this case, Trinity Mini again.

[00:13:31]to improve the query and then send it to the to the document collection to find uh better chunks. Okay, so let's just go and enable this and again it will add a bit of latency because that's another round trip to the model but it's uh it's probably worth it. Okay. Okay, let's illustrate what rewriting does. First, we're going to ask a pretty basic question. Okay. And I've I haven't enabled rewriting. Okay. So, let's see what the model does here. It's going to do rag, but it's going to do rag on that pretty dumb query.

[00:14:16]Okay.

[00:14:18]So, you could argue, well, I do get an answer, right? Um, it's okay. Actually, yeah, it did match documents, etc., etc. uh it's not super detailed. So now let's run the same uh we'll use MMR again but this time we'll add rewriting okay and ask the same thing.

[00:14:42]So first we rewrite the query okay second we send the query to the knowledge base and third we generate the the answer. Okay, so here it's written at net zero carbon emissions definitions. Okay, which is a little better, right? And you can see the reasoning, right? Uh behind the behind the rewriting here, removing filler words, expanding a little bit on the technical concepts, etc. So that's probably uh that's probably uh a better answer here. Okay. Um, let's try yet another one.

[00:15:27]Let's try MMR rewriting.

[00:15:33]Explain what net zero means to the industry to the car industry.

[00:15:46]Let's see.

[00:15:50]Let's see if writing gets a little more creative here.

[00:16:01]Oh, that's very interesting. Okay, so it's rewritten as net zero automotive sector emissions blah blah blah vehicle life cycle which are all good things. And so now I'm actually matching the good stuff, right?

[00:16:17]I'm actually matching the good stuff.

[00:16:20]All those terms are high value and they're certainly present in um in the do in the document collection. And now I'm getting a better answer here. Right.

[00:16:32]So query writing is a is a pretty cool technique. Okay. Okay. Now let's look at reranking. Okay. Let's look at reranking now. So what does rearranking do? So rearranking adds an extra step to evaluate how well each retrieve chunk matches the query. Okay. So now things go like this. We send the user query to um the uh embedding model to retrieve chunks.

[00:17:07]Then we take each chunk with the initial query and we pass that to a reranking model or scoring. Okay. And once we've scored all the chunks with respect to the user query um we return the top scoring uh chunks to the model for generation. Okay. So it's retrieval first um ranking second with respect to the query to double check that um the the the match is actually good and then third we use the model uh the small language model trinity here for generation. So this one adds a bit of latency again although the the ranking models are very small models. So even though let's say we retrieve 10 chunks, uh ranking the 10 doesn't had a a ton of of extra time. Okay. So let's go and try this one. Okay.

[00:18:12]And uh we'll see uh we'll see a little more activity here.

[00:18:19]Okay.

[00:18:21]So here um we did retrieve chunks, right? I'm only retrieving three. You could retrieve more. And and then we're scoring each chunk with respect to the initial query. And the best one was this one, right? And maybe without reanking it wouldn't be, right? So, uh if you if you tend to retrieve lots of chunks, uh reranking is a good technique that could help surface maybe uh a better top chunk um than the one that was uh initially discovered. Okay. And then of course we can we can enable all those things right. Um so we could uh let me uh clear that discussion here. Uh we could try and put all those things together, right? Why not?

[00:19:10]So now a lot is happening. So first we're rewriting the query. So that goes to Trinity. Then with the rewritten query we hit the document collection with the embedding model. And then for each of the chunk we hit the ranking model uh to rerank the chunks and then we pass the reank chunks back to trinity mini for story generation. Okay. And I mean these are simple queries and and my doc collection isn't uh insanely large and you can see this is um this is still plenty fast right? So we can see the rear right, we can see the retrieve chunks etc etc. Okay. So again um there is no Swiss Army knife set of techniques that works every single time. You have to go and experiment um and um and find you know which ones uh work best. So if you have um if you have an evaluation set for uh for your rack system um this is fairly easy to run automatically and you know enable those different options and see um you know in general uh when you when you get u the the most performance right and some of those uh some of those techniques do have extra hyperparameters so you may want to to explore that a little bit but in any case we can See, um, all of that stuff is running local.

[00:20:43]Um, it's certainly fast enough. Uh, I mean, I wouldn't mind working with this system. Um, and you know, maybe try to cut down on the the Trinity latency.

[00:20:54]Maybe try maybe a Q6 uh model instead of a Q8. But the quality is good. You know, we it's generating nicely. It's uh it's definitely understanding the context, etc., etc. Okay. So, um that's that's pretty cool. And then, like I said, we could filter on on particular documents if we knew, you know, we were looking for uh uh yeah, let's try one maybe. Let's try uh let's clear that stuff and let's go and enable um yeah, let's do hybrid and query writing and reranking and let's take uh yeah by John. Why not? Okay. And let's go and ask what are the net zero initiatives.

[00:21:51]Let's see.

[00:21:56]So now we're doing the same thing as we've done before except we're only uh hitting that particular document. Right?

[00:22:04]So let's see if uh anything happens here.

[00:22:12]And yes we do. This is pretty cool, right?

[00:22:18]We get as our badge on specific information. So obviously we could have gone and queried the full collection but uh you know sometimes you know the information is going to be there or maybe you want to chat with a specific document and uh and you just go and uh and hit that document directly. Okay.

[00:22:38]All right. Uh enough of the demo. Let's uh let me walk you through the the code really quickly and and show you how you could add your own PDFs and your own documents uh to this. Okay.

[00:22:51]All the code is on GitHub. Again, I'll put all the links in uh in the video description.

[00:22:58]And um the main thing you want to know is how do I add my files to this? So, basically, you need uh a PDF folder with all your PDF files, right? And just dump them here. Um and Chroma when the app starts Chroma will detect any new file that hasn't been indexed so far and uh it will you know chunk it and ingest it. So you could do it gradually. You can start with a few files and then you can drop and then you can drop more files uh over time. Okay. Um if you want to clear everything because you messed up your document collection, that's very simple.

[00:23:44]You just clobber this directory here.

[00:23:47]Okay. So if you just delete this which is obviously where chroma index is everything you can see my collection here is about 300 megs right. So just nuke that folder and uh and then when the you restart the app um chroma will just reingest everything.

[00:24:08]When it comes to the code I've tried to keep things organized and and maintainable hopefully. So uh this is all about chroma ingesting etc etc. This is all the retrieval code. So hybrid uh MMR etc etc. Um, and um, and that's the Q&A chain from Lang Chain, right? So, should be fairly uh, fairly easy to understand.

[00:24:44]And I've got a bunch of tests as well if you want to tweak and run the tests again. Okay, that's really what I wanted to show you. Um just you know building on the previous rack chatbot, showing you more advanced techniques, showing you how to work completely local with um the RC Trinity mini model which is uh you know more than capable of dealing with all those uh rag queries and uh and generally to show you that local work is possible and uh you don't need to pay a fortune for uh a model AP. API. Uh you don't need to go to the cloud if you don't want to. Working local is great.

[00:25:28]It has a lot of benefits. All right, my friends. Thank you for watching. I hope you liked it. And until next time, keep rocking.

Up Next

RAG Evaluation with RAGAS: Advanced Retrieval Techniques

@AI-Makerspace

35.4K views•2023-12-04

Building Real-Time ML Pipelines with Feature Stores and MLOps Frameworks

@ODSCAI

5.1K views•2022-02-20

Bypassing Tor Censorship: Bridges and Pluggable Transport Guide

@Coding_ForEveryone

397 views•2024-06-11

Neural Networks Explained: Math, Layers, and Learning Fundamentals

@3blue1brown

21.9M views•2017-10-05

Related Study Plans & Knowledge Roadmaps

Structured learning paths in Artificial Intelligence

View All Artificial Intelligence Topics

artificial-intelligence

Building Recommenders: Two-Tower & Vector DBs

01Foundations of Python & Machine Learning Basics

02Introduction to Recommender Systems

03Deep Learning & Representation Learning

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

Latent Diffusion: PyTorch, UNet & CLIP

01Deep Learning & PyTorch Foundations

02Convolutional Networks & Autoencoders

03The Math and Mechanics of Diffusion Models

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

NeRFs: 3D Reconstruction & Instant-NGP

01Math & Python Foundations for 3D Vision

02Photogrammetry & Camera Pose Estimation

03Neural Radiance Fields (NeRF) Core Concepts

+ 2 more learning modules

Explore Roadmap

artificial-intelligence

TFT Forecasting: Energy Demand & Deep Learning

01Programming & Data Science Foundations

02Introduction to Machine Learning & Time Series

03Deep Learning & Sequential Models

+ 3 more learning modules

Explore Roadmap