Deploying Ollama on Kubernetes: A Step-by-Step AI Model Serving Guide

Persistent volumes backed by high-performance shared file systems (such as Azure NetApp Files) cache model weights, eliminating the need for new nodes to re-download hundreds of gigabytes of model data during restarts or scale-out operations. Read-write many access allows all nodes to simultaneously access cached weights, improving deployment efficiency and reducing startup time.

Model caching addresses the fundamental inefficiency of remote model loading by pre-downloading models to local storage. KServe handles inference services by creating Kubernetes deployments with GPU-accessible pods. Without caching, a 7TB LLM requires 15 minutes of startup time (12 min download + 2 min image + pod startup + GPU loading). With caching, this reduces to approximately 1 minute by loading from local disk. The architecture uses a Kubernetes controller that watches model cache resources and creates download jobs on specified nodes. Persistent volumes are then created and attached to inference pods. A node agent continuously monitors local disks and recreates download jobs if models are lost due to node failures, ensuring cache persistence across maintenance events.

Persistent Volume Claims (PVCs) in Kubernetes provide a way to maintain state across pod restarts and failures. When running AI models that require large datasets, PVCs ensure that data persists even when individual pods are terminated. The PVC acts as a claim on storage resources, allowing pods to access the same underlying data regardless of which specific instance is running. This is essential for model serving where the model weights and associated files must remain accessible between different container instances.

Model PVC Provisioning is a Kubernetes deployment strategy that separates AI model storage from container images by using Persistent Volume Claims (PVCs) to store models externally, addressing the limitations of embedding large models directly in images such as slow CI/CD pipelines, large image sizes, and deployment complexity; this approach enables faster model version switching through environment variable changes while reducing infrastructure burden and network consumption.

Volumes are essential Kubernetes resources that provide persistent storage for containers, solving the problem of data loss when containers crash, restart, or are deleted. Persistent Volume (PV) is a cluster-level storage resource pre-defined by administrators, representing actual physical storage. Persistent Volume Claim (PVC) is a user request for storage resources, specifying how much storage is needed and which PV should be claimed. PV YAML configuration includes API version (v1), kind (PersistentVolume), metadata (name), spec with capacity (storage size), accessModes (readWriteOnce), and hostPath (local path on the node). PVC YAML configuration includes API version (v1), kind (PersistentVolumeClaim), metadata (name), spec with accessModes and resources (request for storage size). Pod configuration includes containers with volumeMounts specifying where volumes should be mounted inside the container, and volumes referencing the PVC to connect the pod to storage.

NGINX can be deployed as an authenticating proxy using OAuth 2.0 authentication. The implementation involves: (1) Installing and deploying NGINX as a reverse proxy; (2) Configuring NGINX for OAuth 2.0 by connecting to identity providers like Google, GitHub, or organizational providers; (3) Using /auth endpoint for authentication with error redirects to 401 status; (4) Using proxy_pass to forward authenticated requests to the Kubernetes API server. The API server must be configured with flags --proxy-client-ip and --extra-headers=X-Remote-User,X-Remote-Group,X-Remote-Users to trust the proxy. TLS must secure communication between NGINX and the API server to ensure identity information remains confidential.

To secure Kubernetes Ingress with TLS, you must create a Kubernetes Secret containing the TLS certificate and private key (base64-encoded), then reference this secret in the Ingress resource's TLS block by specifying the hosts and the secret name. The Ingress controller handles SSL/TLS termination, meaning the certificate is managed at the ingress level rather than within individual pods.

Ingress is a Kubernetes resource that manages external access to services within a cluster, providing routing capabilities based on hostnames and paths. Unlike LoadBalancer services that create separate external IPs for each service (costly at scale), Ingress allows multiple services to share a single IP address. Ingress Controllers (like Nginx, HAProxy) watch for Ingress resources and configure the underlying load balancer to route traffic accordingly. For TLS security, three termination modes exist: SSL Passthrough (encrypted traffic passes through without decryption, fastest but least secure), SSL Offloading (load balancer decrypts traffic, reducing application load but compromising security), and SSL Bridging/Re-encrypt (load balancer decrypts, inspects, and re-encrypts traffic, providing the best security).

This section covers advanced security features for Kubernetes ingress. Mutual TLS (mTLS) requires clients to present valid certificates before accessing protected resources, preventing unauthorized access even with stolen credentials. OIDC (OpenID Connect) extends OAuth 2.0 for user authentication with components including Client ID, Client Secret, Authorization Endpoint, Token Endpoint, and JWK URI. Grant types include Authorization Code (web apps), Password (simple login), and Client Credentials (machine-to-machine). App Protect provides comprehensive application security including WAF protection against OWASP Top 10, API protection, Blacklist/Whitelist filtering, sensitive data protection, bot detection, and threat intelligence with signature-based attack detection. Deployment requires creating Signature Set, App Protect Policy, and Log Config, combined into a Virtual Server Policy attached to Virtual Server or Ingress resources.

This video demonstrates how to secure Kubernetes web applications using Single Sign-On (SSO) by configuring Traefik as an Ingress Controller with forward-auth middleware, which delegates authentication to OAuth2 Proxy. OAuth2 Proxy handles login via Keycloak as the identity provider, using encrypted session cookies to remember user sessions. After successful authentication, OAuth2 Proxy passes user headers (including user ID, email, and authorization tokens) to the backend application, ensuring only authenticated users can access the web application.

Deploying large language models at scale requires understanding that no single LLM serving framework is optimal for all scenarios; the best approach depends on specific use cases, model sizes, latency targets, and workload patterns, with key optimization techniques including attention backend selection (Flash Attention vs. Flash Attention Free), speculative decoding, key-value caching, and efficient weight loading, while production deployment demands custom benchmarking, proper GPU resource sharing, horizontal autoscaling with cache-aware metrics, and API gateway integration for security and governance.

Two primary software options exist for running LLM APIs: Text Generation Inference (TGI) from Hugging Face and VLLM. TGI supports Flash Attention and Flash Decoding for long contexts (over 8,000 tokens), and offers on-the-fly quantization (8-bit EQ, 4-bit BNB/NF4) that reduces VRAM requirements by half or four times. VLLM offers OpenAI-style API compatibility, better AWQ (Activation Aware Quantization) support, and pre-quantized model loading. The choice depends on context length requirements, quantization needs, and API compatibility preferences.

vLLM implements end-to-end LLM serving with distributed execution using Megatron-LM for model parallelism and Ray for cluster management. The system achieves 24x throughput improvement over naive Hugging Face usage and 3.5x over TGI, supporting 95% of popular LLMs. Production deployments demonstrate 50% GPU reduction with 2-3x throughput increases, serving thousands of daily conversations. The project has become an industrial standard, with major frameworks adopting page attention. Key capabilities include simple offline batch inference APIs and OpenAI-compatible server deployment. Future work includes speculative decoding and latency optimizations while maintaining throughput focus.

Deploying LLMs is challenging due to instability, safety concerns, and hallucinations. Key challenges include guard rails for safety, feedback integration, licensing issues, and adversarial attacks (prompt injection). Tools like Hugging Face text generation inference, vLLM, Triton Inference Server, and LangChain help with deployment.

TensorRT-LLM and vLLM are leading frameworks for optimizing LLM inference performance. TensorRT-LLM creates GPU-specific optimized CUDA instructions through engine-building, offering substantial performance gains especially on H100 GPUs, with excellent inflight batching capabilities. vLLM provides simpler developer experience. Both frameworks address making high-performance model serving accessible without deep GPU expertise. Model serving frameworks like Truss simplify packaging Python code with Transformers into Docker containers for deployment on platforms like Base 10, abstracting distributed infrastructure complexity.