Classifier-Free Guidance Diffusion Models Explained

The score function is defined as s(x) = ∇_x log p(x), representing the direction of maximum increase in log-probability density. For Gaussian distributions, s(x) = (x - μ)/σ². In diffusion models, the ground truth noise ε_t is a scaled version of the score: ε_t = (1 - ᾱ_t) * s(X_t) / √(ᾱ_t). This reveals that diffusion models are fundamentally score-based generative models, where training implicitly learns to estimate the score function of the data distribution. The loss function performs score matching on the data.

The score function is defined as the gradient of the log probability density function of a distribution. Score-based generative models learn this score function instead of directly modeling the probability distribution, which avoids the need to compute intractable normalization constants. This approach provides a natural way to address the normalization challenge faced by traditional likelihood-based generative models.

The score function is defined as the gradient of the log probability density with respect to the input data: s(x) = ∇ₓ log p(x). Intuitively, the score indicates how to adjust the current point x to increase its likelihood under the data distribution. For a mixture of Gaussians, the score points toward the nearest mode, guiding movement toward regions of high probability density. This property makes score functions useful for sampling and generative modeling.

Score-based models, also known as diffusion models, represent a state-of-the-art class of generative models for images, video, speech, audio, and other continuous data modalities. Unlike likelihood-based models that work with probability density functions requiring normalization constraints, score-based models work with the gradient of the log density (the score function), which is a vector-valued function. The score function can be visualized as a vector field where each arrow indicates the direction to follow to increase log likelihood most rapidly. Energy-based models define flexible probability density functions using arbitrary neural networks to model energy, then renormalizing by dividing by the partition function. Score matching provides an alternative training approach that avoids computing the partition function by matching score functions between model and data through the Fisher Divergence objective. Score-based models directly model the score function (vector-valued function) rather than energy or likelihood. The learning objective is to minimize the Fisher Divergence between the model's score and the true data score. Training requires evaluating the score function efficiently and computing the trace of the Jacobian matrix, which is computationally expensive and scales poorly with dimensionality. Denoising score matching addresses computational challenges by estimating the score of data perturbed with Gaussian noise rather than the original data. By adding noise, the resulting distribution becomes smoother and easier to model. When noise level is small, the noisy distribution approximates the original data distribution. The gradient of the Gaussian transition kernel can be computed in closed form, allowing the objective to be rewritten in terms of this gradient, which is much simpler to compute. The practical algorithm involves: sampling a mini-batch of data points, adding Gaussian noise to each, and computing the denoising score matching loss. The noise level (sigma) must be chosen carefully—a small sigma makes the noisy distribution close to the original data but increases loss variance, while a large sigma makes the problem easier but provides a worse approximation. The model learns to predict the noise that was added to the data, which is a relatively easy task for neural networks.

The score function is defined as the gradient of the log probability density with respect to the data point (∇ₓ log p(x)). In diffusion models, the noise predictor is mathematically equivalent to learning this score function up to a scaling factor, revealing that noise prediction fundamentally estimates gradients of log probabilities. The twisted formula enables computing expected latent means from observed data: E[μ|x] = x - α∇ₓ log p(x). When both prior and likelihood are Gaussian with linear means, the marginal remains Gaussian, enabling closed-form analysis. This theoretical foundation connects forward and reverse processes by showing how to derive latent expectations from observations, providing mathematical justification for score-based generative modeling approaches.

Generative algorithms like VAEs, GANs, and diffusion models all input random noise and convert it to images. When generating images without any prompts or conditions, it is called unconditional image generation where the algorithm outputs any image according to the input noise. However, when training with prompts and label information to generate specific images, these are called conditional generative models.

Conditional generation uses Bayes' rule to compute P(X|C) = P(C|X)P(X)/P(C), where C is a condition (like a class label). This allows generating samples conditioned on specific conditions. The relationship between conditional and unconditional generation is fundamental: unconditional generation is equivalent to generating from the marginal distribution P(X), while conditional generation uses the conditional distribution P(X|C).

Unconditional generation produces samples from a fixed distribution (always getting dogs, for example). Conditional generation uses a variable y (like a prompt) to condition the generation process, producing samples from a conditional distribution P_data_given_y. Most practical applications focus on conditional generation where the model generates objects based on specific inputs or prompts.

There are two types of diffusion models: unconditional and conditional. Unconditional diffusion models generate images that look like their training data without any additional input (e.g., trained on celebrity faces, they only generate celebrity faces). Conditional diffusion models are guided by additional information like text, where training data includes both images and associated text. This is how text-to-image models like DALL-E 3, Midjourney, and Stable Diffusion work.

Optimized diffusion scheduling reduces generation steps from thousands to hundreds by re-timing the denoising process, spending more steps on later stages when images are nearly clean. Classifier guidance uses trained classifiers (like ImageNet classifiers) to guide diffusion toward specific classes by providing gradients at each step. Classifier-free guidance eliminates separate classifiers by training with and without conditioning, then taking the difference between guided and unguided outputs as the guidance signal. Amplifying this signal (e.g., by factor of 10) produces more distinctive images, though it may cause pixel values to exceed valid ranges.

Classifier guidance combines an unconditional diffusion model with an external classifier. The classifier provides a direction in input space that increases confidence in a particular class label. During sampling, the model combines this classification direction with the denoising direction from the diffusion model. Mathematically, this corresponds to Bayesian inference: combining P(X) (unconditional) with P(C|X) (classifier) yields P(X|C) (conditional). The guidance scale γ controls how strongly the classifier influences sampling, effectively sharpening the classifier distribution.

Diffusion models can be guided toward specific classes using either classifier guidance (which uses a separate trained classifier to perturb the denoising trajectory with gradients scaled by a guidance parameter γ) or classifier-free guidance (which trains a single diffusion model with conditioning dropout to learn both conditional and unconditional generation simultaneously, eliminating the need for an external classifier while achieving comparable or better results).

Diffusion models work by starting with noise-free images and progressively adding noise until all information is destroyed, then learning to reverse this process through iterative denoising. Mathematically, they learn the score function (gradient of log-probability density) and follow it while perturbing with noise to sample from the target distribution. Classifier guidance enables test-time control by combining the diffusion score with the gradient of a classifier P(Y|X), effectively sampling from the intersection of the prior and desired properties. This allows generating images compatible with specific conditions without retraining the model.

This section presents the mathematical formulation of classifier guidance in diffusion models. The video explains that the conditional reverse process is proportional to the unconditional reverse process, with the proportionality constant determined by the classifier's gradient with respect to the noisy image. The video shows that the conditional process follows a Gaussian distribution with a mean shifted by the classifier guidance. This mathematical formulation provides a theoretical foundation for understanding how classifier guidance works, allowing the model to generate images that match the desired class while maintaining the diversity of the diffusion process.