Running Ollama on Docker: A Comprehensive Setup Guide

Local AI is an open-source project launched in March that provides a free alternative to OpenAI's cloud-based APIs. It acts as a drop-in replacement for OpenAI's API, allowing users to run LLMs locally or on-premises with consumer-grade hardware. Large language models vary dramatically in size: GPT-3.5 has 175 billion parameters requiring 700GB of memory, while Llama models range from 7-13 billion parameters requiring only 3.5-25GB. This size difference determines whether models can run locally on laptops versus requiring cloud infrastructure. The Transformer architecture transforms language input into output through neural network layers, with the completion API providing conversation history and asking the model to complete the text.

This section compares two primary approaches for deploying LLMs. Local deployment using Hugging Face Transformers allows free, unlimited use with full control over the model, ideal for experimentation and resource-rich environments. The Inference API provides remote model access without local downloads, useful for testing and resource-constrained scenarios, though with usage limits. OpenAI API offers commercial access to premium models like GPT-3.5 Turbo with high-quality responses but requires payment. The choice depends on budget, resource availability, quality requirements, and deployment needs. The video concludes by previewing the next session on RAG (Retrieval-Augmented Generation) for building domain-specific question-answering systems.

Local deployment of large language models offers significant advantages over cloud services, including complete data privacy, cost savings, and offline functionality. LM Studio serves as a user-friendly desktop environment for managing local LLMs across Windows, Mac, and Linux. Models are categorized by functionality: audio-to-text, image-to-text, document analysis, and text-to-image. Model selection involves balancing parameter count (e.g., Llama 8B vs. Gemma 13B) against hardware capabilities. Specialized models like CodeLlama excel at specific tasks despite fewer parameters. Key configuration parameters include temperature (0-1 scale controlling creativity) and response format settings.

A Large Language Model (LLM) is an intelligent system with extensive knowledge across many subjects, capable of coding, answering questions, and helping developers become more productive. When choosing between cloud-based LLMs (OpenRouter, GPT, Gemini) and local deployment, factors include data privacy, cost control, and sovereignty requirements. Cloud models offer larger, more intelligent models with economies of scale, while local deployment provides data privacy and cost control. Open-weight models have improved significantly and now perform comparably to cloud models from less than a year ago.

This section presents a comparative analysis of cloud-based versus local approaches to running large language models. It covers: (1) Cloud execution using VPS services ($30/month), demonstrating that even large models (14B parameters) can run acceptably on CPU-only infrastructure for non-real-time applications; (2) Performance benchmarks comparing different execution environments, including CPU-only versus GPU-accelerated setups; (3) The trade-off between cost and performance, showing how cloud solutions offer lower upfront investment but potentially higher ongoing costs; (4) The importance of context length reduction as a technique to improve performance on resource-constrained systems; (5) Recommendations for different use cases, emphasizing that local GPU-accelerated execution remains superior for real-time applications while cloud solutions suffice for batch processing and cost-sensitive deployments.

Docker volumes are the preferred mechanism for persisting data generated by and used by Docker containers, decoupling storage from containers so data persists even when containers are deleted; volumes can be created using 'docker volume create', listed with 'docker volume ls', inspected with 'docker volume inspect', and removed with 'docker volume rm', and they can be attached to containers using the '-v' flag to share data between containers or mount host directories (bind mounts) for direct file system access.

By default, containers start fresh each time with a pristine filesystem. Persistent storage is achieved using mounts: volumes (managed by Docker daemon, suitable for production and cloud environments) and bind mounts (map host directories into containers, useful for development). Use '-V' or '--mount' syntax with either a volume name or host path.

Docker volumes and bind mounts solve the problem of container ephemeral nature by providing persistent storage solutions; bind mounts directly link a host directory to a container directory, while volumes are Docker-managed logical partitions that offer better lifecycle management, support external storage systems like S3 or NFS, and can be easily created, inspected, and removed using Docker CLI commands, making volumes the preferred choice for most persistent storage needs in containerized applications.

Containers are ephemeral—data inside is lost when containers stop. For databases and stateful services, use volumes for persistent storage. Two methods: (1) Map host paths: 'volumes: - /var/lib/mysql:/var/lib/mysql' creates a host directory for data, (2) Use named volumes: 'volumes: - mysql_data_test:/var/lib/mysql' lets Docker manage storage. Stateless services (web servers) don't need volumes, but stateful services (databases) require them. Verify persistence by stopping and restarting containers—data should remain.

Persist data with named volumes created via docker volume create, then mounted with -v or --mount. Bind mounts map any host directory to container paths. --mount is preferred over -v for explicitness. Choose between Docker-managed volumes and direct host paths based on your persistence requirements.

Proper GPU resource allocation is essential for containerized GPU workloads. The correct workflow involves allocating GPU resources first using Slurm's srun command, then entering the container shell, and finally running the code. Without GPU allocation, containers run on CPU only and are extremely slow. The demonstration shows the dramatic difference in execution speed when GPU resources are properly allocated, with the same code running orders of magnitude faster when executed on GPU resources versus CPU. This highlights the importance of understanding cluster resource management when deploying containerized workloads on HPC systems.

CUDA programming approaches range from C++ (requiring manual memory management) to Python wrappers (PyCUDA, Numba, cuPy) that simplify development. CUDA versions have different features per version, and managing multiple versions requires environment isolation tools like Mamba. Different GPU architectures support different CUDA version ranges (e.g., Ada Lovelace supports CUDA 8+). The CUDA SDK provides low-level driver API, high-level runtime API, and libraries like cuDNN for deep learning. ResNet 50 is a convolutional neural network from the ResNet family, released in 2015, excellent for image classification. The model size is approximately 100MB, making it efficient enough to run on hardware from 2016. Despite being older, it remains popular due to good performance-to-efficiency ratio. It can detect objects like dogs in photos. The Nvidia Container Toolkit enables GPU-accelerated containers, including container runtime, CLI, CDI hooks, and container library. It allows running containers with GPU access using 'docker run --runtime=nvidia --gpus all'. GPU selection methods include using --gpus flag with 'all' or specific device numbers, or using CDI specification with 'device=nvidia.com/gpu=all' (recommended). WSL 2 supports Nvidia GPU acceleration with the latest Windows driver. CUDA becomes available within WSL 2 when the Windows driver is installed, and the CUDA driver is stubbed into WSL 2. Users should not install Nvidia drivers within WSL 2 as this can overwrite WSL 2 drivers. The CUDA toolkit for Linux x86 must be installed separately. Requirements include WSL kernel version 4.19.121+, Maxwell GPUs not supported, and Nvidia Container Toolkit for Docker 19.03 for GPU support.

This section explains why containers solve VM-based GPU provisioning problems. VM approaches have three issues: (1) users must install NVIDIA drivers per VM, (2) GPU monitoring requires CPU binding preventing standard management, and (3) environment conflicts cause application problems. Container-based solutions address these by: (1) providers install drivers once on hosts, (2) containers don't require CPU binding enabling standard monitoring, and (3) images can be consolidated and shared. NVIDIA Container Toolkit (nvidia-docker) enables GPU usage by wrapping Docker commands and handling driver/CUDA injection. It uses labeled files to manage version compatibility between container images and host drivers, preventing errors when drivers are too old for container code.

To enable GPU access in Docker containers, first verify CUDA drivers are installed using nvidia-smi, then install Docker and the Nvidia Container Toolkit, and finally run containers with the --gpus all flag to access GPU resources for accelerated computations.

Dynamic Resource Allocation (DRA) provides an alternative to count-based GPU requests by giving vendors full API control. Key abstractions include resource classes (defined by cluster admins) and resource claims (used by end users), along with class parameters and claim parameters for custom vendor APIs. Users create resource claim templates pointing to GPU resource classes, then reference these templates inside pod specs with local names. This separation enables natural GPU sharing across containers within a pod or across multiple pods, addressing the limitation of not being able to control how oversubscribed GPUs are shared.