Bayesian Inference Example: Univariate Gaussian with Gaussian Prior

Extending Bayesian inference to multivariate Gaussians requires generalizing conjugate priors to matrix-valued distributions. For mean inference with known covariance, the posterior remains Gaussian with updated precision matrix. For precision matrix inference with known mean, the Wishart distribution serves as conjugate prior, with posterior updates W_n = W₀⁻¹ + S_μ and ν_n = ν₀ + n. The inverse-Wishart distribution serves for covariance matrix inference. These multivariate extensions maintain the same mathematical structure as their univariate counterparts, with Wishart reducing to gamma distribution when dimensionality d=1. The framework enables systematic Bayesian analysis of high-dimensional Gaussian data.

This segment extends Bayesian Gaussian estimation to multivariate cases. For data with unknown mean vector μ and known covariance matrix Σ, the posterior mean is a weighted combination of prior mean μ₀ and maximum likelihood estimate. The posterior covariance is (nΣ₀⁻¹ + Σ⁻¹)⁻¹, where Σ₀ is the prior covariance. The matrix inverse replaces scalar division. The posterior predictive distribution has mean μₙ and covariance Σ + Σₙ, following the same pattern as the scalar case with matrices replacing scalars.

This section introduces the fundamental transition from univariate to multivariate Gaussian distributions. The univariate Gaussian is parameterized by mean μ and variance σ², with a bell-shaped PDF f(x) = (1/√(2πσ²))exp(-½(x-μ)²/σ²). Maximum likelihood estimation yields μ̂ as the empirical mean and σ̂² as the average squared deviation. The multivariate extension applies to vector-valued random variables X ∈ ℝ^d, parameterized by mean vector μ ∈ ℝ^d and covariance matrix Σ ∈ ℝ^(d×d). The PDF maintains the exponentiated quadratic form: f(X) = (1/(√(2π)^d√det(Σ)))exp(-½(X-μ)ᵀΣ⁻¹(X-μ)). In two dimensions, constant probability contours form ellipses centered at μ, with Σ determining both spread and orientation.

The multivariate Gaussian extends the univariate case to d-dimensional feature vectors. Its probability density function involves the mean vector μ and covariance matrix Σ. Using matrix calculus similar to the univariate derivation, the maximum likelihood estimates are: the mean vector estimate is the sample mean vector (component-wise average), and the covariance matrix estimate is (1/n)Σ(x_i - μ̂)(x_i - μ̂)ᵀ. Like the univariate case, this gives a biased estimate of the covariance matrix.

The multivariate Gaussian distribution extends the univariate Gaussian to multiple dimensions. Instead of a single mean μ and variance σ², it uses a D-dimensional mean vector μ and a D×D covariance matrix Σ. The covariance matrix contains the covariance between every pair of random variables (X_i and X_j) in the vector. This matrix is symmetric because covariance(X_i, X_j) = covariance(X_j, X_i), and it is positive semi-definite, ensuring it has a valid inverse and non-zero determinant.

For exponential family distributions, conjugate priors have the form: P(η|ν, μ) ∝ exp(ν η^T μ - A(η)ν), where ν is prior sample size and μ is prior mean. The likelihood for N data points is: P(x|η) ∝ exp(η^T Σ U(x_i) - N A(η)). The posterior is: P(η|x,ν,μ) ∝ exp((ν+N) η^T μ' - A(η)(ν+N)), where μ' = (νμ + Σ U(x_i))/(ν+N). The posterior has the same form as the prior with updated parameters.

Conjugate priors exist for all distributions in the exponential family, which includes most famous distributions used in practice (Bernoulli, Binomial, Multinomial, Gaussian). When modeling a new problem, the workflow is: (1) model the data distribution, (2) check for a conjugate prior, (3) if not, consider approximations. Gaussian assumptions are common for continuous data, while Bernoulli/Binomial/Multinomial are used for discrete data.

A conjugate prior is a prior distribution that, when multiplied by the likelihood function, produces a posterior distribution of the same family type. This mathematical convenience simplifies Bayesian calculations. Examples include: Normal likelihood with normal prior (self-conjugate); Binomial/Bernoulli likelihood with beta prior; Exponential/Poisson likelihood with gamma prior; Uniform likelihood with Pareto prior; Multinomial likelihood with Dirichlet prior. When choosing conjugate priors, the posterior retains the same functional form as the prior, making analytical solutions tractable without requiring complex numerical integration.

A conjugate prior is a prior distribution that, when multiplied by the likelihood, produces a posterior of the same functional form. This enables Bayesian inference through simple parameter updates rather than complex integrations. Every exponential family has a conjugate prior that is itself an exponential family. The conjugate prior has the form p(w|α, ν) = exp(α · φ(w) - F(α, ν) + H(w)), where α represents pseudo-values and ν is a pseudo-count. Bayesian inference then reduces to adding sufficient statistics of the data to the prior parameters: α_posterior = α_prior + φ(data) and ν_posterior = ν_prior + n_data.

For exponential family distributions: (1) First moment E[T(x)] = ∇A(θ), second moment Cov[T(x)] = ∇²A(θ); (2) Maximum likelihood depends only on sufficient statistics: ∇A(θ) = (1/n) Σ T(x_i); (3) Conjugate priors have form π(θ) ∝ exp(η₀^T T(θ) - ν₀ A(θ)), updating as η_n = η₀ + Σ T(x_i), ν_n = ν₀ + n. This makes Bayesian computation tractable for exponential family models.

Gaussian process regression offers multiple levels of practical utility: visualization through mean curves and error bars, and decision-making via loss minimization using the full posterior distribution. The model's power comes from selecting mean and covariance functions based on domain knowledge—periodic kernels enforce periodic patterns, symmetric kernels enforce symmetry. This flexibility generalizes Bayesian linear regression and allows encoding prior beliefs about functional forms, making Gaussian processes one of the most versatile general-purpose tools for regression tasks.

A Gaussian process is a nonparametric Bayesian model that defines a distribution over functions using a mean function and a covariance (kernel) function, enabling flexible regression without assuming a specific parametric form; it is mathematically equivalent to Bayesian linear regression with kernels and also corresponds to an infinitely wide single-layer neural network, making it particularly useful for optimization tasks in small-data regimes where each experiment is expensive, such as robotic gait optimization where the goal is to find optimal controller parameters that maximize performance through sequential Bayesian optimization.

Gaussian Processes (GPs) are probabilistic machine learning models that provide full predictive distributions rather than single-point predictions, enabling uncertainty quantification; they work by defining a prior distribution over functions using a kernel function that measures similarity between inputs, then updating this prior with observed data through Bayesian inference to obtain posterior predictive distributions, where hyperparameters control the flexibility and smoothness of the model through mechanisms like length scale and output scale parameters.

Bayesian linear regression provides a probabilistic interpretation of L1 and L2 regularization: using a Gaussian prior on the coefficients yields Ridge regression (L2 regularization), while a Laplacian prior yields Lasso regression (L1 regularization), demonstrating that these regularization techniques are fundamentally equivalent to Bayesian inference with specific prior distributions.

This section develops the complete framework for GP regression. It contrasts parametric models (fixed functional forms with finite parameters) against nonparametric GPs (infinite parameters integrated out probabilistically). The prediction equations are derived: μₚ = K*^T K⁻¹ Y and σₚ² = K** - K*^T K⁻¹ K*, showing how uncertainty decreases as more data is observed. Hyperparameters (length scale l, signal variance σ²) control function smoothness and scale, learned by maximizing the log marginal likelihood. Different covariance functions produce fundamentally different function classes—squared exponential yields smooth functions, absolute exponential yields Brownian bridges, periodic kernels yield sinusoids. The marginalization property ensures tractability despite infinite dimensions. Practical implementation requires careful initialization and hyperparameter selection, with hyperparameters controlling the wiggliness and vertical scale of predicted functions.

Variational inference is a scalable alternative to Markov Chain Monte Carlo (MCMC) for Bayesian inference that trades exactness for computational speed by approximating the true posterior distribution with a simpler parametric form (such as a multivariate normal) and optimizing its parameters to minimize the Kullback-Leibler divergence from the true posterior; this approach is particularly valuable for large datasets, real-time applications, and initializing MCMC samplers, though it sacrifices some precision in characterizing uncertainty compared to exact sampling methods.

MCMC (Markov Chain Monte Carlo) and Variational Inference (VI) are two major approximation methods in Bayesian statistics for estimating posterior distributions. MCMC is theoretically exact in the asymptotic sense—it converges to the true posterior as the number of iterations approaches infinity—but remains practically approximate because we must stop at finite computation. VI, however, remains biased even with infinite computation because it optimizes an approximation objective rather than directly targeting the true posterior. The choice between them depends on the specific problem: VI is preferred when computational budget is tight and approximate answers suffice (e.g., during model development), while MCMC is better suited for final analyses requiring precise uncertainty quantification. Both methods require careful tuning of parameters like warm-up phase length and number of chains, and understanding their trade-offs is essential for effective Bayesian modeling.

Variational inference is a deterministic method for approximating posterior distributions in probabilistic models by optimizing a variational distribution Q to minimize the Kullback-Leibler divergence from the true posterior P, using Jensen's inequality to derive the Evidence Lower Bound (ELBO) as the optimization objective; unlike MCMC methods like Gibbs sampling which are stochastic and require many iterations, variational inference typically converges faster (in dozens of iterations) but requires more complex mathematical derivations, with the mean-field approximation assuming independence between latent variables to simplify computation.

Variational Bayesian inference is a general method for approximating complex posterior distributions in Bayesian models. Unlike exact Bayesian methods that only work for conjugate models, variational methods can be applied to non-conjugate models, which are more complex and commonly encountered in practice. The method approximates the true posterior with a simpler distribution from a chosen parametric family, enabling Bayesian inference in models where exact computation is intractable. When the sample size is much larger than the number of parameters, the Bayesian approach degenerates to maximum likelihood estimation. Variational Bayesian inference transforms the Bayesian inference problem, which involves computing integrals over high-dimensional parameter spaces, into an optimization problem. The Kullback-Leibler (KL) divergence is used as a measure of dissimilarity between the approximate posterior and the true posterior, and minimizing KL divergence is equivalent to maximizing a variational lower bound on the log marginal likelihood. Stochastic gradient estimation allows us to compute gradients of functions that involve expectations by using Monte Carlo sampling. The reparameterization trick is a technique for computing stochastic gradients when the distribution parameters depend on the optimization variables. Instead of sampling directly from the distribution, we express the sample as a deterministic function of the parameters plus noise. This allows us to move the derivative operator inside the expectation, enabling gradient computation. The REINFORCE algorithm is a stochastic gradient descent method for optimizing functions that involve expectations over distributions. The variational lower bound on the log marginal likelihood can be decomposed into two terms: the expected log-likelihood of the data under the approximate posterior, and the negative KL divergence between the approximate posterior and the prior.

Variational Inference (VI) is an optimization-based alternative to Markov Chain Monte Carlo (MCMC) for Bayesian inference, where instead of sampling from the posterior distribution, VI fits a simpler approximating distribution (such as a Gaussian) to the true posterior by minimizing the Kullback-Leibler divergence, trading exactness for computational efficiency and scalability; this approach enables Bayesian Neural Networks and handles large-scale data problems more effectively than traditional MCMC methods, though it introduces bias and may underestimate posterior variance.