Implementing Denoising Diffusion Probabilistic Models in PyTorch

Learning models are fundamentally divided into discriminative and generative types. Discriminative models learn by discriminating between classes, extracting features that distinguish data sets and learning decision boundaries for classification. Generative models, in contrast, learn to model the entire data distribution rather than just differences between classes. A generative model G trained on data X from unknown distribution D produces new samples from a latent distribution, aiming to create a generated distribution D-bar close to the true distribution D according to statistical distance measures.

Traditional machine learning focuses on discriminative tasks—predicting outputs from inputs through feature extraction. Generative modeling reverses this: instead of predicting, the model learns to create new data samples that match the training distribution. Variational Autoencoders (VAEs) attempted this by regularizing latent spaces with KL divergence, but forced all inputs into similar distributions, causing blurry reconstructions. GANs offer an alternative by using adversarial competition between generator and discriminator networks, allowing the generator to produce diverse, realistic samples without explicit distribution matching.

Generative modeling aims to learn the joint distribution between data and labels (or just the data distribution), enabling the model to generate new data samples. Discriminative modeling aims to learn the boundary between classes given the data, determining labels based on observations. The key principle is 'what I understand I should be able to create.' Generative models should be able to sample new data points that look like real data and assign probabilities to existing data points. Maximum likelihood training maximizes the probability of observed real data under the model.

This comprehensive section establishes the theoretical foundations of generative modeling. It begins with the distinction between supervised learning (predicting known target values from labeled data) and unsupervised learning (working with unlabeled data like clustering). The manifold assumption is introduced as a key concept: high-dimensional data actually lies on a lower-dimensional manifold embedded in the higher-dimensional space, allowing us to find meaningful representations by projecting data to this compact space. The section explores latent space properties, including how it supports linear operations like interpolation and analogy, and how metrics in latent space carry semantic meaning. The critical distinction is made between dimensionality reduction algorithms (which can only perform the forward transition from data to latent space) and true generative models (which must handle both encoding and decoding transitions). The section contrasts discriminative models (which predict class probabilities given input features and are limited to their training manifold) with generative models (which predict entire objects like images, making them fundamentally more complex). Bayes' formula is presented as the mathematical foundation: P(X|Y) = P(Y|X) × P(X) / P(Y), decomposing generative modeling into three components. The section concludes with autoencoders (the simplest generative model) and their limitations, followed by variational autoencoders (VAEs) which model the latent space as a probability distribution rather than a single point, enabling generation in regions where no training data exists.

Discriminative modeling involves a discriminator that identifies or classifies content (e.g., determining if an image is real or AI-generated). Generative modeling involves a generator that creates new content (images, videos, audio, text). These two approaches work in competition: the generator creates content while the discriminator evaluates it, providing feedback to improve the generator's output.

Gaussian noise follows a normal distribution characterized by mean and variance values. When adding Gaussian noise to an image, pixel values are slightly modified according to this probability distribution. Closer values to the original have higher probability than distant values. In multi-pixel images, this creates a multi-dimensional probability distribution space where each pixel's value represents a dimension.

The noising process is constructed as a Markov chain where each state represents a noisy version of the data. The chain starts from the data distribution and transitions through progressively noisier states. The transition densities specify how to move from one noisy state to the next, and the chain is designed so that after sufficient steps, the distribution converges to a simple base distribution (typically Gaussian noise).

Gaussian distribution (normal distribution) is a symmetric probability distribution about the mean, appearing as a bell-shaped curve. It is important because most natural phenomena follow this distribution, such as student heights (few very short, few very tall, most in middle range) and exam scores (few zero marks, few perfect scores, most in mid-range). AWGN (Additive White Gaussian Noise) Channel is a fundamental communication model where noise is added to signals during transmission. The noise is 'additive' (added to signal), 'white' (equal power at all frequencies), and 'Gaussian' (normally distributed with zero mean and unit variance).

A Gauss-Markov process is a stochastic process that is both Markovian and Gaussian. The Markov property means the state at time t depends only on the state at time t-1, not on earlier states. The Gaussian property means the state distribution is Gaussian. The state evolution is given by: state(t) = state(t-1) + corruption, where the corruption is Gaussian noise. This process is also known as a Gauss-Markov process because it satisfies both properties simultaneously.

A stochastic process is a function X defined on R × Ω (time × sample space) that associates a real number to each (t, ω) pair. For any fixed time t, X(t, ·) defines a random variable. A Markov chain is a sequence of random variables X_n (n ∈ ℕ) with values in a countable set J. The Markov property states that the probability of future states depends only on the current state, not on the entire history: P(X_n = j | X₀ = i₀, ..., Xₙ₋₁ = iₙ₋₁) = P(X_n = j | Xₙ₋₁ = iₙ₋₁). Transition probabilities pᵢⱼ⁽ⁿ⁾ represent the probability of moving from state i to state j in n steps, defined recursively by pᵢⱼ⁽⁰⁾ = δᵢⱼ and pᵢⱼ⁽ⁿ⁺¹⁾ = Σₖ pᵢₖ⁽ⁿ⁾ pₖⱼ.

The U-Net architecture for DDPM incorporates time-conditioned processing through sinusoidal positional embeddings. Key components include: (1) input/output channels matching data dimensions (1 for MNIST), (2) downsampling blocks increasing features while decreasing spatial resolution, (3) upsampling blocks restoring resolution while reducing features, (4) skip connections preserving spatial information, and (5) time embedding injection via learnable MLP. Each block applies convolutions with batch normalization and ReLU, then adds time-dependent features. This architecture enables the model to condition its predictions on the diffusion time step.

DDPM (Denoising Diffusion Probabilistic Models) is the foundational paper that introduced diffusion models to the broader AI community. It presents the mathematical framework for diffusion models, including the forward and reverse processes, the loss function, and the training procedure. DDPM uses a UNet architecture for the denoising network and demonstrates that diffusion models can achieve competitive performance with GANs while providing better sample diversity.

The UNet architecture serves as the backbone for diffusion models, featuring an encoder-decoder structure with hierarchical processing. Down blocks progressively reduce spatial dimensions while increasing channel depth, incorporating ResNet blocks with residual connections and self-attention mechanisms. Each ResNet block contains two convolutional layers with group normalization and activation functions, connected by residual connections. Following this, self-attention layers process spatial relationships across all pixels. Mid blocks maintain constant resolution through stacked ResNet and attention layers. Up blocks reverse the downsampling process using transposed convolutions, incorporating skip connections from corresponding down blocks. Time embeddings are fused through projection layers in each ResNet block. The complete UNet class integrates all components: initial convolutions match input dimensions, down blocks progressively refine features, mid blocks maintain resolution, and up blocks restore spatial dimensions while reducing channels. Final normalization and convolution return outputs to input dimensions, enabling the model to predict noise at each diffusion time step.

This section covers the DDPM class structure and neural network architecture. The DDPM class initializes beta schedule, alpha values, and alpha_bar. The U-Net architecture consists of an encoder (downsampling path) and decoder (upsampling path) with skip connections. MyBlock is a basic building block with two convolutions and activation functions. Sequential modules chain layers for network construction. The network takes noisy images and time embeddings as input to predict noise (epsilon_theta).