Run Local LLM Coding Assistant in VSCode with Ollama and Continue

Engineers are using large language models (LLMs) to handle boring or difficult coding tasks such as understanding APIs, diffing code, trimming and refactoring code, writing documentation, learning new programming languages, building applications, writing throwaway code, and breaking down technical requirements, with the common theme being that LLMs lighten cognitive load and save time by taking over routine tasks that engineers don't want to do or find mentally challenging.

Large Language Models (LLMs) are sophisticated AI systems trained on massive amounts of text data using supervised learning and reinforcement learning techniques, enabling them to generate coherent text by predicting the next word in a sequence; they represent a subset of generative AI within the broader field of artificial intelligence, which traces its origins to Alan Turing in the 1940s and the Dartmouth Conference of 1956, and these models have significant ecological impacts due to their high energy consumption during training and operation.

Large Language Models (LLMs) are neural networks trained on massive text data to understand and generate natural language, differing fundamentally from traditional programming by learning patterns from examples rather than following explicit instructions; they work through tokenization (splitting text into units), embeddings (converting tokens to numerical vectors representing semantic meaning), and transformer architectures with attention mechanisms that enable context-aware predictions, though they face challenges including bias, hallucinations, high computational costs, and limitations in mathematical reasoning despite their remarkable capabilities in tasks like translation, summarization, and creative writing.

Large Language Models (LLMs) are generative AI algorithms that work specifically with text data, taking text as input and generating text as output. The term 'large' refers to the massive amount of data these models are trained on. LLMs primarily work on text data, but code is also considered text, so LLMs can generate code. Omni-modal models (or multi-modal models) can handle multiple types of input and output, taking text, images, or audio as input and generating text, images, or audio as output. Generative AI is a broader category that includes LLMs as one of its components, encompassing any algorithm that can generate new content including text, images, audio, and video.

Large Language Models (LLMs) are the core technology behind generative AI. An LLM takes user input (prompt) and generates output (text, code, images). LLMs work by predicting the next word in a sequence based on patterns learned from training data. Unlike traditional language models that predict the next word from a few words, LLMs are trained on vast amounts of internet data including books, websites, code repositories, and articles. This extensive training enables them to generate sophisticated content, answer complex questions, and perform tasks like code generation and translation. The scale of training data and computational resources required are what make LLMs 'large' and capable.

The terminal (command line interface) is used to install applications by pasting installation commands. Unlike typical website downloads where you click a link to download, some software requires terminal commands. To use the terminal, search for it in your operating system's search bar and open it. Once opened, you can paste the installation command and press enter to execute the installation process.

The command line interface (CLI) allows direct interaction with the computer without using a graphical interface. Key concepts include: (1) Package managers like Homebrew act as command-line app stores, (2) The 'cd' command changes directories, (3) The 'ls' command lists directory contents, (4) The 'mv' command moves or renames files. Understanding CLI is essential for advanced software installation and configuration.

The Command Line Interface (CLI) is a direct method to interact with computers, offering faster and more powerful control compared to graphical user interfaces; essential commands include 'ls' for listing files, 'cd' for changing directories, 'mkdir' for creating folders, and 'touch' for creating files, while avoiding spaces in filenames due to their special character status in CLI syntax.

The terminal (Command Line Interface or CLI) provides direct system access through text commands. Unlike graphical interfaces (GUI) used in browsers and phones, the CLI requires typing commands. The shell (software managing CLI) accepts user input and executes actions. Commands follow a grammatical structure: Command (verb/action), Options (adverbs modifying behavior, starting with dashes), and Arguments (objects the command acts upon). Commands are typically abbreviated English words (ls for list, cp for copy, mkdir for make directory).

The terminal or command line is an application that runs on computers and serves as an interface where users can type text commands. It allows users to navigate files and folders, create files, change or update things, install and run programs. Many programming tools require interaction via text commands in the command line rather than graphical interfaces.

VRAM is the critical bottleneck for local AI, not CPU or GPU processing speed. All model parameters must reside permanently in VRAM for processing. When models exceed VRAM capacity, data offloads to system RAM or SSD, creating severe bandwidth bottlenecks that cause system unresponsiveness. VRAM requirements scale with model size: 7B parameter models need 6-8GB, 14B models need 10-16GB, and 30-70B models require 24+GB. The 8GB VRAM limitation trap created by the gaming industry has become a dead end for AI workloads. Modern GPUs with only 8GB VRAM are fundamentally limited for local AI, functioning only at absolute limits with basic models. The RTX 3060 with 12GB VRAM has become extremely valuable because those extra 4GB make the difference between successful model execution or system failure.

Cloud AI services run models on powerful remote servers, while local AI downloads models to your computer for offline use. This eliminates subscription costs, protects code privacy, and allows use without internet. VRAM (Video RAM) on your graphics card determines which models you can run locally. Larger models require more VRAM - a 7B parameter model needs 6-8GB, while 700B models require hundreds of gigabytes. Models exceeding your VRAM capacity will run extremely slowly or not at all.

For AI inference, VRAM matters more than system RAM. A gaming PC with 128GB RAM and RTX 5080 (16GB VRAM) cannot run large models because GPU VRAM is the bottleneck. The only real consumer options are MacBooks with unified memory (128GB RAM costs $3,000+), DGX Spark (12 TPS but not useful), or Strix Halo laptops. The RTX 5090 (32GB VRAM) is comparable to an A100 enterprise chip but still cannot run large models due to VRAM limitations.

Running local AI models on your PC depends on two key hardware metrics: VRAM (video memory on your graphics card) and RAM (system memory). VRAM is critical because AI models prefer fast GPU memory for optimal performance. To find your VRAM, open Task Manager, go to Performance > GPU, and check 'dedicated GPU memory.' If no GPU is listed, your VRAM is effectively zero, and models will run on slower system RAM. Model size is measured in billions of parameters (B), with approximately 0.5 GB of memory needed per billion parameters at normal quality settings. For best performance, models should fit entirely within VRAM; otherwise, they spill onto slower system memory, causing significant slowdowns. Tools like LM Studio and online compatibility checkers help determine which models will run well on your specific hardware before downloading.

When building a PC for local AI agents, VRAM (video memory) is the single most critical specification to prioritize over raw processing power, as it determines how much of the AI model can fit in the graphics card's memory; models like 7B, 14B, 32B, and 70B require approximately 5GB, 10GB, 20GB, and 40GB of VRAM respectively, and the system's overall performance depends on having sufficient VRAM to load the entire model without swapping to slower system memory, which causes dramatic slowdowns from smooth operation to unusable speeds.