Academic Databases
Explore Our Semantic Databases
STEM & Tech
Math, Physics, CS & Engineering...
Economics
Macro, Micro, Econometrics...
Business
Management, Marketing, Finance...
Health & Life Sciences
Biology, Medicine, Neuroscience...
Humanities & Social Sciences
Psychology, Law, Philosophy, History...
General Studies
General & interdisciplinary studies...
Explore Topics
Features

Implementing Stable Diffusion VAE in PyTorch: A Step-by-Step Guide (50 chars)

Added: 2026-08-07

Deep Dive

Proficiency in PyTorch, including constructing custom neural network modules, defining training loops, and utilizing autograd.

Семинар. Модули в PyTorch

Семинар. Модули в PyTorch

24.3K

В PyTorch можно создавать кастомные нейронные сети, наследуясь от класса nn.Module, где необходимо переопределить метод forward() для определения прямой прогонки данных, а backward() автоматически вычисляется на основе архитектуры сети; для обучения требуется функция потерь (например, cross-entropy для классификации), оптимизатор, и важно использовать nn.Parameter() для создания обучаемых параметров, которые будут автоматически отслеживаться моделью.

Click to watch →

Lesson 5: Deep Learning 2019 - Back propagation; Accelerated SGD; Neural net from scratch

Lesson 5: Deep Learning 2019 - Back propagation; Accelerated SGD; Neural net from scratch

88.8K

PyTorch requires subclassing nn.Module to create custom layers, implementing __init__ for parameters and forward for computation. Linear layers perform affine transformations: output = X × W + b. Modules have parameters() returning all learnable parameters (weights and biases). Data loaders handle batching, shuffling, and GPU placement automatically. Training loops iterate through data loaders, compute predictions, calculate loss, backpropagate gradients, and update parameters. This abstraction handles computational details while exposing the core training workflow.

Click to watch →

Building makemore Part 3: Activations & Gradients, BatchNorm

Building makemore Part 3: Activations & Gradients, BatchNorm

464.3K

PyTorch builds neural networks using modular components: Linear layers (torch.nn.Linear), BatchNorm1d, Tanh, and custom modules. Each module contains parameters and implements forward() methods. Training loops iterate through data, perform forward passes, compute loss, retain gradients, execute backward passes, and apply optimizers. This modular structure enables building complex architectures by stacking predefined layers.

Click to watch →

Training Deep Neural Networks on a GPU | Deep Learning with PyTorch (3/6)

Training Deep Neural Networks on a GPU | Deep Learning with PyTorch (3/6)

17.8K

To define a custom neural network in PyTorch, extend the nn.Module class and implement two required methods: __init__ (constructor) and forward. In __init__, define the layers (e.g., self.linear1 = nn.Linear(input_size, hidden_size), self.linear2 = nn.Linear(hidden_size, output_size)). In forward, process the input through each layer sequentially, applying activation functions between layers. For example: def forward(self, xb): xb = xb.view(-1, 784); xb = F.relu(self.linear1(xb)); return self.linear2(xb).

Click to watch →

Introduction to PyTorch

Introduction to PyTorch

318.7K

PyTorch is an open-source ML framework accelerating research to production with deep learning primitives, GPU acceleration, and libraries for vision/NLP/audio. The framework enables fast iteration through regular Python without DSLs. Tensors are core abstractions representing multi-dimensional arrays with over 300 operations, created as 32-bit floats by default but supporting custom data types. AutoGrad enables automatic differentiation through computation history tracking, allowing backward() to compute gradients for any path including branches and loops. Models inherit from torch.nn.Module with __init__ for layer construction and forward() for computation flow, assuming batched input with batch dimension as first axis.

Click to watch →

Deep Dive

Fundamental concepts of standard Autoencoders (AEs), specifically encoder-decoder architectures, bottleneck layers, and reconstruction loss.

Autoencoders - EXPLAINED

Autoencoders - EXPLAINED

79.7K

Autoencoders are unsupervised neural network architectures consisting of three main components: an encoder, a bottleneck (latent layer), and a decoder. The encoder compresses input data into a lower-dimensional representation, the bottleneck stores this compressed latent representation, and the decoder reconstructs the original input from the latent code. The output has the same dimensions as the input, and the objective is to minimize reconstruction loss through techniques like backpropagation.

Click to watch →

Lecture 28 : Autoencoder

Lecture 28 : Autoencoder

18.5K

An autoencoder is an unsupervised learning algorithm for representation learning that encodes input data into compressed representations. Unlike supervised learning with labeled data, autoencoders reconstruct input from encoded form. The architecture consists of an encoder (input to hidden layer) and decoder (hidden layer to output). A bottleneck layer with fewer nodes than input forces compressed representation, preventing identity mapping. The network removes data correlation and redundancy, preserving only essential features for reconstruction.

Click to watch →

Autoencoders | Deep Learning Animated

Autoencoders | Deep Learning Animated

123.3K

The basic architecture of an autoencoder consists of three main components: (1) The encoder compresses input data into a 'latent-space' representation, which is a low-dimensional space capturing essential features; (2) The bottleneck (or latent space) layer, which is the smallest layer in the network, holds this compressed representation; (3) The decoder reconstructs the original input data from the compressed representation produced by the encoder and bottleneck.

Click to watch →

Autoencoders Tutorial | Autoencoders In Deep Learning | Tensorflow Training | Edureka

Autoencoders Tutorial | Autoencoders In Deep Learning | Tensorflow Training | Edureka

1K

Autoencoders consist of three main components: Encoder (compresses input to latent space), Code/Latent Space (compressed representation), and Decoder (reconstructs input from latent space). Four hyperparameters define training: code size (compression level), number of layers (depth), loss function (MSE or cross-entropy), and nodes per layer (symmetric encoder-decoder structure). The bottleneck approach balances compactness (storage efficiency) against information retention about relevant variables, forcing the model to identify essential data characteristics.

Click to watch →

Introduction To Autoencoders In Machine Learning.

Introduction To Autoencoders In Machine Learning.

15K

An autoencoder consists of three main components: (1) The encoder compresses input data into a lower-dimensional representation by stripping away irrelevant details; (2) The bottleneck (or latent space) is a narrow layer that forces the network to condense information into its essential form; (3) The decoder reconstructs the original input from the compressed representation. This architecture mimics human ability to identify core concepts despite superficial variations.

Click to watch →

Deep Dive

Theoretical understanding of Variational Autoencoders (VAEs), including the reparameterization trick, latent distribution modeling, and Kullback-Leibler (KL) divergence.

[Variational Autoencoder] Auto-Encoding Variational Bayes | AISC Foundational

[Variational Autoencoder] Auto-Encoding Variational Bayes | AISC Foundational

33.7K

This section presents the technical solution enabling VAE training: the reparameterization trick. By expressing latent variables as deterministic functions of parameters and independent noise (Z = g(Φ, ε)), gradients can flow through the sampling operation. For Gaussian distributions, this means Z = μ + σ × ε where ε ~ N(0,1). When priors and posteriors are both Gaussian, KL divergence has a closed-form solution, eliminating Monte Carlo estimation for this term. The complete loss combines this analytical KL term with Monte Carlo estimates of reconstruction error. Large mini-batches allow setting Monte Carlo samples L=1 for efficiency, balancing computational cost against estimation quality.

Click to watch →

Variational Autoencoders Theory Explained | Generative AI Course

Variational Autoencoders Theory Explained | Generative AI Course

15.2K

Variational Autoencoders (VAEs) address the limitations of vanilla autoencoders by mapping each input to a multivariate normal distribution in the latent space rather than a single point, enabling continuous sampling and interpolation; the VAE architecture uses an encoder that outputs both a mean vector (μ) and a standard deviation vector (σ), which are combined with a random epsilon sample through the reparameterization trick to generate latent representations, while the loss function combines reconstruction loss (to preserve input information) and KL divergence (to ensure the latent distribution matches a standard normal prior), creating a continuous latent space where nearby points produce similar decoded outputs.

Click to watch →

Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU)

Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU)

35K

Helmholtz Machines, developed in 1995, consist of a generative model and recognition model, but the wake-sleep algorithm proved unreliable. Variational Autoencoders (VAEs) define generative processes through cascades of stochastic layers with conditional probability distributions, introducing stochasticity at each layer to generate diverse samples. VAEs define a variational lower bound (ELBO) that separates log-likelihood from KL divergence between approximate and true posteriors, enabling tractable optimization. The reparameterization trick enables efficient training by expressing Gaussian sampling as deterministic transformation plus noise injection: z = μ(θ) + σ(θ) × ε. This pushes gradients inside expectations, transforming stochastic optimization into deterministic backpropagation. Before this trick, reinforcement learning-based approaches suffered from high variance. The reparameterization trick dramatically reduces training variance and enables learning of otherwise intractable models. VAEs allow generation of diverse samples from latent codes, enabling tasks like text-to-image generation where a single sentence produces multiple plausible images rather than just one deterministic output.

Click to watch →

Variational AutoEncoder Paper Walkthrough

Variational AutoEncoder Paper Walkthrough

16.7K

A Variational Autoencoder (VAE) extends traditional autoencoders by introducing probabilistic elements to enable sampling from a latent space, solving the problem of intractable posterior distributions through variational inference. The key innovation is the reparameterization trick, which allows sampling from a distribution while maintaining differentiability by expressing the latent variable as a deterministic function of parameters plus independent noise (e.g., Z = μ + σ × ε for Gaussian distributions). This enables efficient optimization using stochastic gradient descent, where the loss function balances reconstruction error (how well the decoder recreates the input) with a KL divergence regularizer (encouraging the latent space to follow a standard normal prior).

Click to watch →

Understanding Variational Autoencoders (VAEs)

Understanding Variational Autoencoders (VAEs)

56.4K

Variational Autoencoders (VAEs) are generative models that learn the underlying distribution of data by mapping input data to a lower-dimensional latent space through an encoder network, then reconstructing data from this latent representation using a decoder network; they overcome the intractability of directly computing the marginal likelihood by using variational inference to approximate the posterior distribution, maximizing a lower bound called the Evidence Lower Bound (ELBO) which balances reconstruction accuracy against regularization of the latent space to ensure smooth, continuous, and complete representations.

Click to watch →

Deep Dive

Deep learning for computer vision, specifically Convolutional Neural Networks (CNNs), residual blocks, and spatial downsampling/upsampling techniques.

Deep Learning for Computer Vision (Andrej Karpathy, OpenAI)

Deep Learning for Computer Vision (Andrej Karpathy, OpenAI)

180.4K

Convolutional Neural Networks (CNNs) are specialized neural networks designed to exploit the local connectivity structure of grid-like data such as images, videos, and spectrograms. Unlike fully connected networks that connect every neuron to all inputs, CNNs use convolutional layers with small spatial filters that slide across input volumes, sharing weights across spatial positions to detect local features efficiently. This weight-sharing assumption dramatically reduces the number of parameters (from millions to thousands) while maintaining performance. Pooling layers then downsample these feature maps to control computational complexity. The breakthrough came in 2012 with AlexNet, which demonstrated that scaling up CNN architectures and training them end-to-end on large datasets like ImageNet could dramatically outperform traditional hand-crafted feature approaches, achieving human-level performance on image classification tasks. Subsequent advances like VGGNet, GoogleNet, and Residual Networks further improved performance by optimizing architectural choices, with Residual Networks solving the vanishing gradient problem that prevented training of very deep networks. CNNs have since become foundational building blocks for diverse applications including object detection, segmentation, captioning, and reinforcement learning.

Click to watch →

Лекция. Архитектуры CNN

Лекция. Архитектуры CNN

32K

Skip connections (residual connections) solve vanishing gradients by adding direct paths between non-adjacent layers. Implementation: x3 = f(x2) + x1, where f represents intermediate layers. This allows gradients to flow through direct paths while learning through intermediate layers. ResNet combines CNNs with skip connections, using residual blocks with convolutions, batch normalization, and ReLU. Variants include ResNet18, 34, 50, 101, 152, differing in depth and block types.

Click to watch →

Lecture 9 | CNN Architectures

Lecture 9 | CNN Architectures

466K

ResNet (2015) revolutionized deep learning by enabling training of 152-layer networks, achieving 3.57% top-5 error (better than human performance). The key innovation is residual connections: each layer learns a residual mapping F(x) rather than direct mapping H(x), with output computed as H(x) = F(x) + x. This design allows easy learning of identity mappings (F(x) = 0) and provides direct gradient paths. For deeper networks, bottleneck layers (1x1 convs) reduce computational cost. ResNet uses batch normalization, SGD with momentum, and learning rate decay. It won all ImageNet classification and detection contests in 2015.

Click to watch →

In-Depth Look: ResNet Architecture and Residual Block Explained

In-Depth Look: ResNet Architecture and Residual Block Explained

36.3K

Residual Neural Networks (ResNets) address the vanishing gradient problem in deep neural networks by using residual blocks that learn the difference between input and output (instead of the direct mapping), enabling the construction of extremely deep networks with hundreds or even thousands of layers; these blocks typically consist of two convolutional layers with batch normalization and ReLU activation, followed by an addition operation that adds the original input to the transformed output, and ResNet versions 1 and 2 differ in whether the activation function precedes or succeeds the addition operation.

Click to watch →

CS231n Winter 2016 Lecture 7 Convolutional Neural Networks LxfUGhug iQ-sHyIqu_S5Ks.mp4

CS231n Winter 2016 Lecture 7 Convolutional Neural Networks LxfUGhug iQ-sHyIqu_S5Ks.mp4

68.8K

ResNet (2015) solved the problem of training very deep networks by introducing residual blocks with skip connections. A 152-layer ResNet achieved 3.57% top-5 error on ImageNet. The key insight is that very deep plain networks suffer from vanishing gradients and optimization difficulties, causing training error to increase with depth despite having more capacity. Residual connections allow gradients to flow directly through identity shortcuts, enabling training of hundreds of layers. ResNet uses residual blocks where each block learns a residual function (what to add to the input) rather than a direct mapping. The block contains two or more convolutional layers, and the input is added to the output of these layers. This formulation makes it easier to optimize deep networks because the identity function serves as the default behavior. Skip connections allow gradients to flow unimpeded through the network, solving the vanishing gradient problem. Current trends include eliminating pooling layers in favor of strided convolutions, using smaller filter sizes (especially 3×3), increasing depth significantly, and removing fully connected layers through global average pooling. Beyond image classification, CNNs power applications like AlphaGo's Go-playing system, demonstrating their versatility in processing structured spatial data and making strategic decisions.

Click to watch →

VAE Basics

Architecture & Loss

Self-Attention

Attention Block

Encoder Layers

Decoder Design

Training Setup

Loss & Optimizer

Stable Diffusion

Integration Result

VAE Basics

0:00

Playing Section

1
Explains VAE as a dimensionality reduction tool for image generation.
2
Contrasts autoencoders' fixed latent vectors with VAE's probabilistic distributions.
3
Highlights VAE's ability to generate new variations from learned latent pools.

Deep Dive

Integrating the implemented VAE into a complete Latent Diffusion Model (LDM) pipeline to enable latent-space denoising.

2 Amazing Ideas in Latent Diffusion Models LDM w/ VAE, U-Net & CLIP: Generative AI #stablediffusion

2 Amazing Ideas in Latent Diffusion Models LDM w/ VAE, U-Net & CLIP: Generative AI #stablediffusion

2.2K

Latent Diffusion Models (LDM) are a class of generative AI models that run the diffusion process in latent space instead of pixel space, significantly reducing training costs and improving inference speed. The architecture combines three key components: a Variational Autoencoder (VAE) for perceptual compression, a U-Net for semantic manipulation and denoising, and a text encoder like CLIP for conditional generation. By first compressing pixel-level redundancy through the VAE and then performing the diffusion process on the learned latent representations, LDM achieves efficient high-resolution image synthesis while maintaining flexibility for conditional inputs such as text prompts.

Click to watch →

How does Stable Diffusion work? – Latent Diffusion Models EXPLAINED

How does Stable Diffusion work? – Latent Diffusion Models EXPLAINED

Latent Diffusion Models address computational challenges by abandoning direct work in image pixel space and instead operating in a lower-dimensional latent space. The process involves encoding images through an encoder (such as a VQ-VAE) to obtain compressed representations, then applying diffusion noise to these compressed codes. The decoder then reconstructs the final image from the denoised latent representation, significantly reducing computational requirements.

Click to watch →

Диффузионные модели простым языком: как генерируются изображения и видео

Диффузионные модели простым языком: как генерируются изображения и видео

114

Latent diffusion is the same diffusion process but operates in a compressed latent representation rather than the original image space. This is done using a Variational Autoencoder (VAE). The process involves: (1) Encoding the original image through the VAE to get a compressed latent representation (e.g., 512x512 becomes 64x64), (2) Performing diffusion in this latent space, (3) Decoding the latent representation back to an image. This approach saves memory and speeds up both training and inference. The latent representation is typically 4-channel and 8 times smaller in spatial dimensions.

Click to watch →

Diffusion Models (DDPM, DDIM, Stable Diffusion, U-Net) Deep Dive By Google Engineer | Multimodal LLM

Diffusion Models (DDPM, DDIM, Stable Diffusion, U-Net) Deep Dive By Google Engineer | Multimodal LLM

6.3K

Guided diffusion enables controlled image generation by incorporating additional embeddings (such as text queries) into the UNET alongside noisy images and time embeddings. During training, the model learns to recognize these conditioning signals through self-attention blocks. Latent Diffusion Model (LDM) improves efficiency by performing diffusion in a lower-dimensional latent space. VQ-VAE (Vector Quantized VAE) differs from regular VAEs by using a discrete latent space represented by a learned codebook of K embedding vectors. The quantization process involves finding the closest matching codebook vector and replacing the continuous representation. Three main benefits: more meaningful latent codes, avoidance of posterior collapse, and higher computational efficiency. LDM training consists of two phases: training the VQ-VAE encoder-decoder, then performing diffusion training in the latent space with conditioning.

Click to watch →

Lecture 9.2 - New Generative Models (CMU Multimodal Machine Learning, Fall 2023)

Lecture 9.2 - New Generative Models (CMU Multimodal Machine Learning, Fall 2023)

1.3K

This section details diffusion model training and latent diffusion. Diffusion models are trained by maximizing the ELBO, which decomposes into reconstruction terms at each time step (how well the reverse process removes noise) and a prior term (ensuring the final noisy representation is close to standard Gaussian). The model learns to predict the noise added at each step. Latent diffusion models (like Stable Diffusion) combine VAEs with diffusion: images are first compressed through a VAE encoder to a lower-dimensional latent space Z, then diffusion is performed in this compressed space (much faster than pixel space), and finally the VAE decoder reconstructs the image. This achieves high-quality generation while being computationally efficient.

Click to watch →

Deep Dive

Studying the U-Net architecture and Cross-Attention mechanisms used to guide the diffusion process with text or image prompts.

Diffusion Models Explained: From Noise to Images (U‑Net Architecture Deep Dive)

Diffusion Models Explained: From Noise to Images (U‑Net Architecture Deep Dive)

104

Modern text-to-image systems consist of: (1) A text-conditioned latent U-Net that performs denoising on compressed representations; (2) A scheduler controlling iterative denoising steps; (3) A text encoder converting prompts into meaningful inputs; (4) A variational autoencoder expanding latent representations back to full resolution. The process starts with random compressed noise, applies denoising guided by text conditioning, then expands the result. Different random seeds produce different variations from the same prompt.

Click to watch →

Stable Diffusion | Stable Diffusion Model Architecture | Stable Diffusion Explained

Stable Diffusion | Stable Diffusion Model Architecture | Stable Diffusion Explained

1.2K

Stable Diffusion is a text-to-image generation model that operates through a sophisticated architecture combining three key components: a Variational Autoencoder (VAE) for image compression and decompression, a U-Net architecture with skip connections for iterative denoising, and cross-attention mechanisms that integrate text embeddings to guide image generation; the model works by first applying noise to an image through forward diffusion, then reversing this process through 20-50 denoising steps where the U-Net predicts and subtracts noise while using cross-attention to ensure the generated image aligns with the text prompt, all operating efficiently in a compressed latent space rather than directly in pixel space.

Click to watch →

Указание условий генерации для диффузионной модели: фиксируем позу, форму и др. (aka ControlNet)

Указание условий генерации для диффузионной модели: фиксируем позу, форму и др. (aka ControlNet)

382

This section explains how cross-attention works for both text and images. For text, it captures relationships between words (like subject-verb relationships). For images, it captures relationships between different regions (like how a penguin relates to an egg in the image). The mechanism is the same: computing attention scores between elements and combining information with learned weights. The speaker emphasizes that this allows the model to capture complex relationships between different parts of the input, whether text or image regions.

Click to watch →

Stable Diffusion from Scratch in PyTorch | Conditional Latent Diffusion Models

Stable Diffusion from Scratch in PyTorch | Conditional Latent Diffusion Models

20.5K

Cross-attention extends self-attention by allowing queries from one modality (spatial features) to attend to keys and values from another modality (text). The mechanism remains similar: queries are projections of feature map cells, while keys and values are projections of text token representations. The text encoder (like BERT) produces sequence-length × embedding-dimension outputs. The context projection layer transforms text embeddings to match the channel dimension. Multi-head attention computes relevance between spatial elements and text tokens, then aggregates the context representation. This enables the diffusion model to condition on text descriptions by attending to relevant words during generation. The same cross-attention mechanism can handle other modalities like image patches.

Click to watch →

Google A.I. Diffusion Image Editing w/ Prompt to Prompt

Google A.I. Diffusion Image Editing w/ Prompt to Prompt

42K

Diffusion models generate images by starting from random noise and progressively refining it based on text prompts. This explains why changing the random seed produces different images from the same prompt. Cross-attention maps reveal how models associate text tokens with image regions—for example, 'cyberpunk' relates to overall style, 'man' to the rider, and 'motorcycle' to both the vehicle and rider. This demonstrates semantic understanding where the model recognizes that a motorcycle depiction includes the rider on top of it, not just the vehicle.

Click to watch →

Deep Dive

Exploring advanced latent representation techniques, such as Vector Quantized VAEs (VQ-VAEs) and VQ-GANs, to understand discrete autoencoding.

VQ-VAEs: Neural Discrete Representation Learning | Paper + PyTorch Code Explained

VQ-VAEs: Neural Discrete Representation Learning | Paper + PyTorch Code Explained

52.4K

VQ-VAE (Vector Quantized Variational Autoencoder) is a generative model that differs from traditional VAEs by using discrete latent representations instead of continuous ones. The model consists of an encoder that maps inputs to continuous latent vectors, a codebook containing learnable discrete vectors, and a decoder that reconstructs outputs. The key innovation is the vector quantization process where encoder outputs are snapped to the nearest codebook vector using L2 norm, creating a discrete latent representation. To enable backpropagation through this non-differentiable operation, the straight-through estimator is used, which copies gradients from the decoder input to the encoder output. The loss function includes reconstruction loss (MSE under Gaussian likelihood assumptions), codebook loss (pushing encoded vectors toward codebook vectors), and commitment loss (pushing codebook vectors toward encoded vectors). After training, an autoregressive model is fitted on the discrete latent codes to enable generation via ancestral sampling. This approach enables efficient compression, avoids posterior collapse, and produces high-quality reconstructions and novel samples, forming the foundation for advanced models like DALL-E, Jukebox, and VQ-GAN.

Click to watch →

CS 198-126: Lecture 9 - Autoencoders, VAEs, Generative Modeling

CS 198-126: Lecture 9 - Autoencoders, VAEs, Generative Modeling

10.1K

VQ-VAEs extend VAEs to discrete settings by adding a learnable codebook of codewords. After Gaussian sampling, latents are quantized to nearest codewords, creating discrete latent positions. Codebook locations are learned alongside network parameters. To achieve combinatorial diversity without exponentially growing codebooks, quantization applies hierarchically—for example, a 32x32 grid where each element independently selects from 512 codewords, yielding 512^(32×32) distinct outputs. This enables discrete generation (text, tokens) while maintaining manageable complexity.

Click to watch →

Vector-Quantized Variational Autoencoders (VQ-VAEs)

Vector-Quantized Variational Autoencoders (VQ-VAEs)

25.7K

VQ-VAEs are a variant of Variational Autoencoders that discretize the latent representation by mapping continuous encoding vectors to a finite codebook through nearest-neighbor quantization, enabling efficient data compression, natural tokenization for downstream tasks like Transformers, and better modeling of inherently discrete modalities such as speech and text; the model uses straight-through estimation to handle the non-differentiable quantization step and incorporates codebook loss and commitment loss to ensure stable training while learning an implicit prior distribution over code words.

Click to watch →

VQ-VAE | Everything you need to know about it | Explanation and Implementation

VQ-VAE | Everything you need to know about it | Explanation and Implementation

28.9K

VQ-VAE is a generative model that learns discrete latent representations by replacing continuous latent variables with a codebook of embedding vectors, addressing the limitations of traditional VAEs where high-dimensional continuous latent spaces struggle to capture meaningful representations for complex data like natural images; the model uses straight-through gradient estimation to handle the non-differentiable quantization operation and incorporates codebook loss and commitment loss to train both the encoder and the discrete embeddings effectively.

Click to watch →

VQ-VAE Explained in 3 Minutes! | How Neural Networks Learn Discrete Representations

VQ-VAE Explained in 3 Minutes! | How Neural Networks Learn Discrete Representations

114

VQ-VAE (Vector Quantized Variational Autoencoder) is a neural network architecture that learns discrete latent representations instead of continuous ones, solving the posterior collapse problem in traditional VAEs by using a codebook where the encoder output is replaced by the nearest embedding vector, with gradients passed through via a straight-through estimator during backpropagation.

Click to watch →

Deep Dive

Implementing perceptual loss (e.g., LPIPS) and adversarial training (discriminators) to improve the visual fidelity of the VAE reconstructions.

Stable Diffusion From Scratch Part 1: Perceptual Loss Functions (LPIPS + PatchGAN)

Stable Diffusion From Scratch Part 1: Perceptual Loss Functions (LPIPS + PatchGAN)

843

Perceptual loss functions like LPIPS improve generative model quality by measuring perceptual similarity between images rather than pixel-wise differences; unlike mean squared error which fails to capture human visual perception (as shown by identical MSE scores for vastly different images), LPIPS uses pre-trained CNNs (such as VGG) to extract meaningful features and compares these features across different image pairs, enabling better reconstruction quality in variational autoencoders and latent diffusion models.

Click to watch →

Stanford CME296 Diffusion & Large Vision Models | Spring 2026 | Lecture 4 - Latent Space & Guidance

Stanford CME296 Diffusion & Large Vision Models | Spring 2026 | Lecture 4 - Latent Space & Guidance

402

This section derives the VAE loss function using maximum likelihood estimation, yielding the ELBO (Evidence Lower Bound) with reconstruction and KL divergence terms. It addresses the fundamental limitation: pixelwise L2 loss produces blurry reconstructions because it penalizes pixel-level differences, causing averaged values. Two strategies combat this: perceptual loss (LPIPS) compares images in feature space using convolutional network features that capture structural information and are spatially invariant; adversarial loss uses discriminators to distinguish real from generated images, incentivizing realistic, detailed outputs. The section concludes with latent space diffusion, where generation models operate in VAE latent spaces rather than pixel spaces. Training involves encoding images to latents, sampling noisy versions, and training flow matching models to predict velocity fields. Inference involves sampling noise, solving velocity fields, and decoding to pixel space. The encoder captures semantic information (making diffusion's task easier), while the decoder adds perceptual details, with the decoder typically larger than the encoder.

Click to watch →

Открытая лекция. Авторегрессионная Генерация Изображений

Открытая лекция. Авторегрессионная Генерация Изображений

111

VQ-GAN improved VQ-VAE by replacing MSE/L1 reconstruction loss with perceptual loss (LPIPS) and GAN loss. LPIPS loss uses pre-trained networks to measure perceptual similarity, producing sharper images than MSE/L1 which tend to produce blurry results. This combination of L1, GAN, and LPIPS losses has become standard practice for training autoencoders.

Click to watch →

Stable Diffusion & Friends: High-Resolution Image Synthesis via Two-Stage Generative Models

Stable Diffusion & Friends: High-Resolution Image Synthesis via Two-Stage Generative Models

5.1K

VQ-GAN improves upon standard VQ-VAE by introducing perceptual loss and adversarial training. While VQ-VAE uses pixel-wise reconstruction losses (L2/L1), these cause blurry reconstructions by encouraging averaging. VQ-GAN adds L_clip_slots (perceptual loss aligned with human vision) and adversarial loss using patch-wise discriminators. An adaptive weighting strategy dynamically balances loss contributions based on gradients, preventing manual hyperparameter tuning. Progressive training first optimizes reconstruction before introducing adversarial components. This combination achieves sharp, realistic reconstructions while maintaining computational efficiency, demonstrating how perceptual objectives outperform purely pixel-based optimization in generative modeling.

Click to watch →

Building SDXL’s AutoencoderKL from Scratch in PyTorch

Building SDXL’s AutoencoderKL from Scratch in PyTorch

141

The decoder implements progressive upsampling using nearest neighbor interpolation followed by two residual blocks per stage, gradually increasing spatial dimensions while refining features. The final convolutional head projects from base channels to output channels (3 for RGB images). Training employs three loss components: L1 reconstruction loss (more robust than L2), perceptual loss using LPIPS (which compares images in VGG feature space to capture textures and semantics), and KL divergence for regularization. The perceptual loss addresses the common VAE problem of blurry reconstructions by encouraging the model to preserve fine details that pixel-wise losses might miss. Proper attention score scaling by dividing by sqrt(d_k) prevents softmax saturation. The reparameterization trick enables gradient flow through latent sampling during training.

Click to watch →

VAE Basics

Architecture & Loss

Self-Attention

Attention Block

Encoder Layers

Decoder Design

Training Setup

Loss & Optimizer

Stable Diffusion

Integration Result

VAE Basics

0:00

Playing Section

1
Explains VAE as a dimensionality reduction tool for image generation.
2
Contrasts autoencoders' fixed latent vectors with VAE's probabilistic distributions.
3
Highlights VAE's ability to generate new variations from learned latent pools.

31.1K views75043:18@freecodecampOriginal Release: 2024-12-04

A Variational Autoencoder (VAE) is a type of unsupervised neural network that extends the basic autoencoder by learning a probability distribution over the latent space rather than producing fixed latent vectors, enabling it to generate new, varied image outputs instead of merely reconstructing training examples; the VAE architecture consists of an encoder that outputs mean and log variance for each data point, a reparameterization step that samples from this distribution to create the latent vector, and a decoder that reconstructs the image, with training involving both reconstruction loss (measuring how well the decoder recreates the input) and KL divergence loss (ensuring the learned distribution stays close to a standard normal prior).

[00:00:00]Learn how to build a stable diffusion VAE from scratch using PyTorch. VAE stands for Variational Autoencoder. It's a type of autoencoder and a neural network that trains using an unsupervised technique. They're widely used in image generation models, mainly on latent diffusion-based and GANs-based image generation models. Machine learning engineer Harsh Bhatt teaches this course. Hey guys in this video we're going to learn how to implement VAE and train a VAE model. VAE is variational autoencoder which has huge impact in image generation model like stable diffusion and DALIN. Let's first take a look at like how VAE works and then we'll go to implement it and then train it. So to understand the VAE we need to first understand the autoencoder as it is the base of it. So autoencoder is an unsupervised learning neural network model. It is primarily used for dimensionality reduction so let's suppose if we want to train our stable diffusion models we cannot feed the 4k images to train the model which approaches like 8 million pixels to train the neural network which will be very computationally expensive. To solve this problem we will need a model which actually reduces the dimensionality of this 8 billion pixels into some fixed small vector with high rich features of those images. That's where the autoencoder comes in the picture which helps us to encode the image into a latent. So latent is a fixed point over the latent space so that's it our problem solved but that is not because autoencoder has some limitations when it comes to generations. So basically it is essentially designed to compress the images into latent and to reconstruct back to the original image so it lacks the ability to generate new and varied outputs. That's where the VAE comes into the picture unlike autoencoder rather than memorizing the patterns. It generates variations of images so the major technical improvement in VAE it does not directly generates the latent. First it generates the probability distribution and then converts it into the latent. So by learning and predicting the probability distribution over the latent space the VAE encodes each data points to not a fixed vector but as a range defined by mean and variance. So let's suppose if VAE is trained on 100 digits so the 7 will be in one pool and the 2 will be in another pool. So every digit have a unique pools in the latent space rather than a fixed vector or fixed point it will it will generate a pool of it and then as there are pools it has the possibility to generate the variations of it. But unlike in autoencoders where you have only a fixed vector for a specific thing and it is exactly based on what model has seen in the training examples. Suppose if we want to generate the number 7 it will going to generate the same close same as close to as the original one what has seen in the training images. The VAE actually learns this by predicting and learning the probability distribution and in the loss function it actually uses the reconstruction loss and the KL divergence which we're going to look at when we're going to implement this. So that's exactly what the VAE is and I hope you understood let's go and implement it. Hey guys so let's take a look at how to implement the VAE and then we're going to train it. So I have prepared a google colab notebook for you so we can run I can run the code at the same time and explain you how exactly it's working. In the VAE architecture you have this input image the input and then you have encoder which basically going to predict probability distribution view and it will give you this two things the standard deviation and the mean and then you can use this to create the latent space right and then you can use this latent space to reconstruct back using the decoder and then after decoding it you have the reconstructed image. Now to train this model you will go and do the backward pass and the loss will be for this is reconstruction loss plus KL divergence. So reconstruction as you already know about that the difference between the original image and the reconstructed image it could be like mean's gradual loss. So the KL divergence basically used to estimate the difference between two probability distribution.

[00:04:55]So we add those two together and then it we have the final loss which we're going to use it to train it and so we'll have like a if you have millions of samples it will go in to train multiple times and then it will become much better. So as this is a very large model you can have like millions of images and it can able to generalize those very well. So first we have imported many modules which is not many actually but like some torches and maths functions a skid learn to have to train to split or NOS definitely. So I have written some like some explanation for you so let's suppose if you want to try together you I would recommend to try together and then you can or you can try it after and then read all the things like what exactly it is saying I have written some notes for you. So it's a good notebook to actually have to learn about VA which I will gonna use to if I need in future. So make sure you have this runtime D4 GPU so we're going to use GPU for this and then you need to run this I don't know why it's taking so long okay it's getting down.

[00:06:21]So let's go ahead and then here we have I have explained about the self-attention self-attention you might probably know if not it's basically give you attention scores of different tokens if you have let's suppose in NLP natural language processing if you are like have sentences it just gives you the attention scores which features which tokens are more important sentence similar to in images or basically in visual models you have pixels so it will give you which pixels are most important give you the scores attention scores which will provide you the all the KQV scores and then you basically it has attention let me show you the attention scores so I think yeah this was a very bad image what the hell wow so attention of attention is all you need so I'm going to go to PDF one so ensure this all you need this will have very good information so yeah this is a formula this is formula actually which gives you the attention score and that's what it is describing here actually like the lines which are dim which has less attention scores which are very dark and bold which are actually have the highest attention score and it goes similar to pixels also right so here we are implementing self-attention self-attention are basically going to used with within the same sequence so let's suppose within the same pixels area so they are like cross attention also which I'm not gonna talk about here but self-attention is basically used for specifically same sequence so let's suppose if you have a sentence of something let's suppose I'm harsh so I'm harsh it will going to do it will divide into tokens of the same sequence and then match it each each other so I think it is actually in a self-attention because the law will never be perfect and the law will never be perfect matching the same tokens to each other right so that's what self-attention is so it goes against the same idea with the images so you have let's suppose you have to imagine they are like pixels and then the same pixel will be here and then to match and have a score of which each other right so that's that's so in init function we have self dot number of heads as we are like doing the same process of so in the init function we have number of heads in projection layer which is the linear layer which goes with dimension of like embedding dimension and three times of embedding dimension as we want to actually divide it into three tensors which like q, k, v you might already know about it so I would recommend to read this paper to understand about this self-attention it has like because it these three things which help us to attain the attention score q, k, and v right and this thing in the root it is the constant so now it's if you want bias is equal to in projection layer it's like true here and then our projection our projection is the output of this self-attention is the embedding dimension and embedding dimension and the bias will be based on the the argument right so and then we have self dot d heads which is like dimension of heads so the dimension of heads will be like the embedding dimension divided by the number of heads we have and the forward middle will have like this thing the x will be like a bad size sequence and dimension the bad size is something we will have this the sequence length in the images and images terms it will going to be the multiplication of the height and width so the let's suppose if we have three by five images that was 15 so 15 it will be the sequence length and dimension will be the number of channels of the images here so we are just extracting from this x the shapes and then we are defining interim shape which is the required shape we need to do to calculate the attention score in parallelly which is the back size sequence length and we are dividing to number of heads and the dimension of heads and then here we are getting we are calling the in projection layer and then we are chunking it out into three different tenses which has a shape of back size sequence length and d embedding which is dimension embedding and then we are changing it to the interim shape that is the required shape to do the to calculate the attention in parallel and then we are like transposing it to attain this shape which is this one earlier we had this um sequence length number of heads just we are like transposing it to just swap it out swap this thing so you had number of heads will come earlier and the sequence that will come later right um yeah so basically the back size number of heads and the the sequence line which is the heights and widths the multiplication of height and width and the number of channels you have you will have okay so now after that we can calculate the attention look to calculate the attention we have weight uh like the first part we are we are calculating the first part which is like q matrix multiply by k dot transpose by transposing it to actually do the to have the right shape for the matrix multiplication so after that if you want to fill mask of the weight then this code will actually do that and weights dot mask will fill the mask and then we have the weight divided by so it's the same formula which we have seen it here divided by dk right this is the same and then um and then the uh and then we are doing the f dot short max function so here we have the soft max and then the last we would multiply it with here we are multiplying it v and then transposing it back to the original one interim shape sequence length number of heads dimension divided by heads and then change back to the uh what it is required to have the output prediction this is bad size sequence length d embedding which was equals to what it has been uh during the what it how it came actually here and then we are using calling the out projection to get the output so that this is the all cell potential and the attention block is basically so this is all about the self-attention and then we have the attention block uh which just uses this self-attention in in the um residual p so basically you probably know about the residual we need to use a residual layers here because residual layers allow us to do the skip connection which help us to to create a deep network so if you don't have residual layer or don't do the skip connections there's a probability like not probability but yeah definitely the model going to be off it with few layers and uh we possibly can't train the very deep large models with large parameters so the large data will not able to generalize that work so this attention block uh so this initializes first group norm and the attention which is a self-attention module which we have defined and then forward we have this residual which we are used to loading the x first and then uh use uh calling grip norm and then then we are changing the shape to actually work with attention uh self-attention module here and then so basically as i describe we need bad size and then the sequence length and the channel so as in the input we have bad size channels and h in the blue rather than red so we are just uh you uh getting the features h multiplied by multiply by w so here i think we are doing it so we are changing the shape here uh x dot view would be like h multiplied by w so we will have just number of bad channels and h and w which is the features here and then we are transposing it so the h and w will become here and the channels so when we have the uh output from the attention we can just uh transpose it back and then in the last we are just adding the residual here uh which should be resistory here yeah yeah okay so um this was the attention block and then we can like i think i've called this one not okay and then i have to call this and then we have the visual block uh specifically for encoder and decoder so here uh i'm initializing the grip norm so normalization layer and then con one so in residual layer we don't actually change the shapes of the input so here also like we haven't changed the shapes in attention layer like the shapes were the same and also the output was the same output shape was the same so in the visual layer the the input and output shape will be the same also so the con one will actually extract the features but it will not change or reduce dimension as because the kernel size will be three and the padding one when the kernel size is three and the padding is one it don't change the shape i think there is formula for that you possibly can search it out to when when you apply con 2d there's a formula that based on in channels and out channels how much the um it reduces how much the shape is going to reduce or the the height width going to be reduced so this will not change the shape neither this one uh so there are two blocks here like the versus the con group norm and con one and group two no group two group norm two and con two right and if there's eight channels out channels are same just define the n and dot identity as a visual layer right so in forward function we have backside in channels and h and w and then we will just uh using the race issue here x dot clone and then dripped calling the group group norm and then there's a like activation function we are calling and the con one and then the second block which is the group norm two and the con two and then in the end we are adding the uh residual layer and x right so then we are moving forward to the encoder so encoder is um more interesting as it has more layers and like it going to actually do the mean work for us to convert into latents so this whole class is basically sequential here so directly defining this in our super dot init and so first we have this con 2d to actually so basically the intention for it is going to be here so so we will basically reduce the dimensions from here so you're going to see a lot of con 2d layers and so the first we have the con 2d layers which actually i've also defined here the shapes like the what was the initial shape and how it going to impact and like what exactly the output shape going to be so here we're not changing the um size of the image but just the channels and then we have um residual block uh so we were going to use more various so we were going to use lot of residual block in between because we need to have a very deep network so we're going to use residual block uh a lot here between the con 2d so then we have another con 2d and then here we have stride 2 so which um has h divided by 2 divided by 2 so half the size of the image now we have and again residual block, residual block and con 2d now we have the divided by 4 size and then we have residual block, residual block so it's kind of same pattern you have till we do this until here until we reach the h divided by 8 like until we reach x divided by 8 w divided by 8 and then uh we can just use the attention block here for doing the attention stuff and then again we uh in the end we have just uh we we can use residual block and then and in the end block of it uh we are using residual block and the group normalization and this uh salu function activation functions and this con 2d layers so the final uh shape will be going to be bad size 8 channels h divided by 8 and w divided by 8 so so we have reduced the size of the image very much and then we can use it to generate the latency and then decode it back in our forward function first we are doing is uh we are padding it so x equals to f dot pad so it going to pad it uh to the right and bottom and then after that we are chunking it the mean and variance which is the more most important part of the va initial uh most important part of the va initial size we had bad size 8 s divided by 8 w divided by 8 and then we are generating the not generating but like chunking it into two tensors which is uh bad size and then we have like four dimensions because we are uh we are chunking it in dimension one so this is going to be half the channel is going to be half so yeah from eight to four and this will going to be the same so mean and log will have the same inner dimensions which like uh h divided by 8 and w divided by 8 and then um then we are doing is we are doing log so this is the log variance so it's not the variance log variance which we going to use to clamp and then we are doing the re-parameterization technique which is uh the process of generating the latent so how we doing is this is the std standard deviation so we you we can use the log variance and then call this dodge dot exponent function to have the to generate the standard deviation and then this eps is basically uh the eps is something i think um wait what is eps yeah i think eps is basically the um so eps is uh so oh oh so eps is uh the constant so it's like dodge or trend line and then this is the formula for the uh generating the latent so the x is our latent which we can also say it like a z and then we are just multiplying with some constant which this constant is coming from like a artificial library uh and in the decoder we can we will going to divide it also this constant so this was all about the encoder and uh now let's take a look at decoder so in decoder we have again this uh so the intention let's take a look at what are the intentions so now we have the latent space the intention for the intention to use the decoder is to actually use this latent space to reconstruct back the original image that's what the decoder uh is going to do so again it's a sequential layer it's going to uh it has various residual layers and contour d layers the main layer in here is upsample so so nn dot upsample which going to upsample by a factor so here it was earlier we had like bath size for 32 by 32 image and then when we're going to call this uh function we're going to upsample it by a factor of 2 because the scale factor is 2 so it's going to be 64 city floor and then we are calling it uh at a certain interval after defining contour d and a screw block and then here we the same scale factor which is 2 we have 128 and then we're going to do this again to buy 256 because our uh input image going to be 256 and output image going to be 256 so and then we actually again using the group norm and then nn dot value activation function uh in forward method first we actually removing the constant what we have multiplied it with x and then we just like iterating it all the models we have in the decoder and we are returning this x which is the output so the output will be going to be the shape will be going to bath size three will be the channel so the the original image in uh channel and the original image height original image at width we are going to return it so that was the decoder and then we can call it i think i haven't called this encoder yet call this one oops encoder and decoder so yeah so this was all about the um va and uh now we can like use these modules to actually train the va so the next part is to download the process data so i have so i already have this data set of docs it consists of images of dogs of various uh breeds and then you have this you can unzip it and here this function we're going to split it into the training and test dir and this class this this is the last model which wraps all the encoder and decoder so we can call this and then the last uh cell we have is and then we have like a train va so here we are basically uh using this va module to actually create a training loop and then to do the back forward and backward pass so let's take a look at what are the constants and how we are defining it so the number of people going to be 100 just like you can tweak it up like to certain level and how much you can train uh learning rate learning rate is like one this is the learning rate is like a typical learning rate for va which i think probably it can converge it better with that but you can tweak it if you want to and then we have beta which is like kl divergence weight so as i mentioned kl divergence is like is used to find a difference or integrate the for the comparison up to uh probability distribution so here we will use this uh in the kl divergence loss this will transform to a specific shape and the transform transforms dot compose function that size will have four you can just tweak it up uh based on your memory uses and um data set uh this this defines data sets and then it's a data loader we are defining data loader from here and then here we have a model using the va module and then the optimizer we're going to be the adam one yeah and then here we have learning it also we are like um using accumulation of step so we have here yeah yeah we are just using accumulation of steps if you don't know about the accumulation of steps i would recommend to a little bit read about it like how this works accumulation of steps it's basically like normalize the loss to account for accumulation so all the like the and here just we are storing the training losses uh so in our training this is our training loop uh now we can train our v now finally okay so just model a train typical thing and then here the and then here we are iterating over the data loader image that uh putting into a device i think device we have what was yeah good advice so and then forward pass it will return deconstructor encoded model so as the model is v which returns the two things decoded and encoded right so so reconstructive which is like a decoder and encoded the latent vector and now we need to compute the loss the reconstruction law is going to be msc loss uh which going to be reconstructed and b compares it with images the original image then extract mean and log variance from the encoded so as we have already mentioned it so so we had the latent so now we are again uh yeah again extracting it mean and log variance uh although like i think you can try it up with uh not again uh splitting up like in so i think here i'm like uh first chucking it into mean and log and then defining it again i think that's the right way to do it i believe and then again i'm uh extracting the mean and log variance but let me know if you read or like uh know that we can do any other way based on what i know we can extract uh again mean and log variance and then use it over the KL divergence laws to calculate the KL divergence so this formula calculates the KL divergence i think you can read more about the KL divergence already formula so so this is the formula actually um so you can you can read more about it possibly uh so this KL divergence we will going to use it uh as it was mentioned here reconstruction plus KL divergence so so the total loss will be equals to reconstruction plus the KL divergence so we are like multiplying with the constant the KL divergence we're going to have a constant multiplied which we have to find here the beta now we have a loss and then now we can do the backward pass here and uh this to the uh optimize.zero grad dot step optimize.step and zero grad and storing the training losses of um this basically i'm just i wanted to see like reconstructed image so at every step i'm like generating this uh reconstructed image reconstructed.py which you can actually try it out during the training you can just uh at every step it will save the image so you can check it out like how what how exactly the eating constructor image looks like uh so i think let's call this and let's see how exactly this is training and then we can check it out the constructed.py dng yep so it started training and yeah and then this is the constructed.png looks like a very small image but i think yeah initially you probably won't able to see anything um but after like training at a certain level you possibly can see the dog which is here so that was the all about the VAE the instrumentation and how to train it hey guys so this is a second part of the video of building and training VAE uh so uh in that video i explained how to build a VAE from scratch and how to create a training pipeline of it and train it so in this video i have trained a VAE uh till 19th epoch so this is my trained model file what i'm gonna do with this is i will gonna use our custom VAE like the VAE we have implemented uh here like the this VAE uh the model we have implemented all the things in core and decoder and attention block and decision block we will going to integrate this into the stable diffusion pipeline and then we will try to generate an image with that that's why i have trained this VAE till 19th epoch which is still very early and uh still not very generalized but it gonna work in the way where we can just recognize whatever we are prompting uh so let's try this um class VAE so this is the our older VAE section but uh for that making compatible with diffusion library and diffusion stable diffusion pipeline to create our custom VAE which has some specific functions so this is a custom VAE and then this encode function this decode function and the forward function will just only return the x reconstructed z is the latent from the encode uh from the encoder and the x reconstructed is the image we have from the decoder this load pre-trained weights will going to load the state dict of the this model which we're gonna use and then we have another module which is a diffuser comparable VAE here we are uh loading uh autoencoder kl which is like a another module torch module which help us to create comparable VAE uh with like diffuser comparable VAE so i'm using her as an inheritance so this diffusion comparable we have all the functionality what autoencoder kl has here i'm just defining cell.va to VAE um and getting the VAE from the constructor and then the encode and decode function uh here some of the things i'm printing just like debug and stuff like that so yeah i think just run this thing and then run this thing and then here i'm uh loading the uh this compvist class table diffusion you can just like i can go here show you what kind of model this is this is a stable diffusion model yeah oh it was released on i think 2022 and uh plus yeah i think this model will work with our VAE it is a 512 by 512 model by running this you can download this in the local DIA here uh dot slash model DIA and then the next you have to do is like you have to uh just check it out like what exactly uh the base structure of it and i have basically tried to look at it okay what other things it has and what other things i need uh to be made compatible like whatever the VAE we have constructor implemented how to make compatible with that autoencoder klva so the next we have is like i'm just importing a torch here i'm loading the our strained custom VAE uh which is like here VAE model repop 19.pth file this is a pts file my application is CPU so in this section i am replacing some typos and stuff so because like there were some missing keys and stuff like that so and it was not very compatible like the our implementation was not compatible to the what we wanted to actually do with uh stable diffusion so i have to actually replace some namings and stuff all the way like the architecture is is same but the namings and stuff is different that's why i'm using this uh which this code will actually replace some names and stuff but the architecture going to be the same and it will add all the uh keys and stuff there right um so this is what it does and then just it just loads the VAE then i can just we can just see uh the architecture of the VAE here and you can just check it out like how it is constructed and match it out like the other VAEs or basically you can load other VAEs like i'm loading this one right now this um a diffusion model you can try to load another diffusion model uh possibly like it should be like 512 by 512 because if we go with STXL something different uh bigger models it won't work so yep and then what we are doing is like we are just loading the stable diffusion pipeline this VAE custom VAE comparable VAE and then inside this we are just putting the VAE and then here stable diffusion pipeline dot from pre-trained the stable diffusion model and the here we have the option to uh integrate the custom VAE or whatever the VAE want to integrate so this is here we are just injecting the comparable VAE and then pi dot 2 and moving to CUDA let's run this and here prompt is our photo of a dog and then the we are running the pipeline with inference step is 50 let's see how this image will look like i'm using this promo photo for dog because our VAE is trained on the dog photos so let's see okay so it does uh look like a dog like you can just recognize it although like the image quality is like very noisy because the VAE is trained uh till only 19 epoch it should like go to like 100 epoch or something like that and like originally VAE is trained till like a huge number of hours uh so we haven't trained that much uh i don't have that much compute at the moment to actually train that but if you want to try please try it and then try to integrate that VAE and then try to see how the image quality is improving and let me know guys how this will work if you want to try this out but that's it for the for this tutorial i hope you enjoyed and understood let me know if you have a question and if you want to see more content like this tutorials subscribe to coded harsh thank you guys for watching this video yeah

Up Next

AI Diffusion Models: How Text Becomes Images

@3blue1brown

1.5M views•2025-07-25

Building Real-Time ML Pipelines with Feature Stores and MLOps Frameworks

@ODSCAI

5.1K views•2022-02-20

Machine Learning and AI Concepts Explained with Animations

@freecodecamp

241.3K views•2025-04-22

Neural Networks Explained: Math, Layers, and Learning Fundamentals

@3blue1brown

21.9M views•2017-10-05

Related Study Plans & Knowledge Roadmaps

Structured learning paths in Artificial Intelligence

View All Artificial Intelligence Topics

artificial-intelligence

Building Recommenders: Two-Tower & Vector DBs

01Foundations of Python & Machine Learning Basics

02Introduction to Recommender Systems

03Deep Learning & Representation Learning

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

Latent Diffusion: PyTorch, UNet & CLIP

01Deep Learning & PyTorch Foundations

02Convolutional Networks & Autoencoders

03The Math and Mechanics of Diffusion Models

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

NeRFs: 3D Reconstruction & Instant-NGP

01Math & Python Foundations for 3D Vision

02Photogrammetry & Camera Pose Estimation

03Neural Radiance Fields (NeRF) Core Concepts

+ 2 more learning modules

Explore Roadmap

artificial-intelligence

TFT Forecasting: Energy Demand & Deep Learning

01Programming & Data Science Foundations

02Introduction to Machine Learning & Time Series

03Deep Learning & Sequential Models

+ 3 more learning modules

Explore Roadmap