Normal Prior Normal Likelihood Posterior Derivation

Bayesian linear regression specifies priors on regression coefficients and error variance. Normal priors on coefficients (mean zero, wide standard deviation) express weak prior beliefs. Half-Cauchy or uniform priors on variance handle scale parameters. The likelihood assumes normally distributed errors around the linear prediction. Sampling algorithms like MCMC generate posterior samples, providing full uncertainty quantification for all parameters simultaneously. This approach naturally extends to multiple predictors and hierarchical structures.

In Bayesian linear regression, when a normal prior is placed on the regression coefficient vector β with fixed error variance σ², the posterior distribution of β remains normal due to conjugacy, with mean equal to (X'X)^(-1)X'y and variance equal to (X'X)^(-1)σ², demonstrating how the normal distribution serves as a conjugate prior for the normal likelihood in multiple regression models.

Multiple linear regression models the relationship between a response variable Y and multiple predictors X₁,...,X_p. The model assumes Y_i = β₀ + β₁X_i₁ + ... + β_pX_ip + ε_i, where ε_i ~ N(0, σ²) are independent errors. In Bayesian terms, we specify: (1) likelihood p(y|β, σ²) as product of normal densities; (2) priors on coefficients (often normal with mean zero) and error variance (often inverse gamma). The posterior combines these to estimate β and σ². This Bayesian formulation makes all assumptions explicit, unlike classical regression which often brushes details under the rug. The conjugate prior structure allows analytical solutions in simple cases, but realistic applications require numerical methods due to high-dimensional integration challenges.

In Bayesian linear regression, when the response vector y given β follows a normal distribution with mean Xβ and covariance σ²I, and β itself follows a normal prior with mean α and covariance B, the posterior distribution of β given y is also normal, demonstrating conjugacy; the posterior mean combines the prior mean α with the ordinary least squares estimator (X^T X)^(-1)X^T y, and the posterior variance is (X^T Σ^(-1) X + B^(-1))^(-1), showing how Bayesian estimation naturally incorporates prior information while converging to classical least squares estimates when the prior becomes diffuse.

A Bayesian linear regression model specifies: (1) Priors on coefficients and intercept (normal distributions), (2) Likelihood function modeling data as normally distributed around predicted values, (3) Error term modeled as normal distribution. The model structure follows: observed data Y ~ Normal(expected_value, σ), where expected_value = intercept + coefficient × predictor. This mathematical specification translates into code using probabilistic programming languages.

A conjugate prior is a prior distribution that, when multiplied by the likelihood function, produces a posterior distribution of the same family type. This mathematical convenience simplifies Bayesian calculations. Examples include: Normal likelihood with normal prior (self-conjugate); Binomial/Bernoulli likelihood with beta prior; Exponential/Poisson likelihood with gamma prior; Uniform likelihood with Pareto prior; Multinomial likelihood with Dirichlet prior. When choosing conjugate priors, the posterior retains the same functional form as the prior, making analytical solutions tractable without requiring complex numerical integration.

For exponential family distributions, conjugate priors have the form: P(η|ν, μ) ∝ exp(ν η^T μ - A(η)ν), where ν is prior sample size and μ is prior mean. The likelihood for N data points is: P(x|η) ∝ exp(η^T Σ U(x_i) - N A(η)). The posterior is: P(η|x,ν,μ) ∝ exp((ν+N) η^T μ' - A(η)(ν+N)), where μ' = (νμ + Σ U(x_i))/(ν+N). The posterior has the same form as the prior with updated parameters.

A conjugate prior is a prior distribution that, when multiplied by the likelihood, produces a posterior of the same functional form. This enables Bayesian inference through simple parameter updates rather than complex integrations. Every exponential family has a conjugate prior that is itself an exponential family. The conjugate prior has the form p(w|α, ν) = exp(α · φ(w) - F(α, ν) + H(w)), where α represents pseudo-values and ν is a pseudo-count. Bayesian inference then reduces to adding sufficient statistics of the data to the prior parameters: α_posterior = α_prior + φ(data) and ν_posterior = ν_prior + n_data.

For any distribution in the exponential family, there exists a conjugate prior distribution that, when multiplied by the likelihood, produces another distribution from the same family. The conjugate prior has the form P(θ|α, β) ∝ exp(α · τ(x) - βA(θ)). This allows Bayesian inference to be performed analytically without requiring complex integration.

This extensive section extends conjugate priors to multi-class and continuous data, then presents the exponential family as the unifying mathematical framework. The Dirichlet distribution generalizes the beta distribution to K categories, serving as the conjugate prior for the multinomial distribution. The sufficient statistics are category counts, and the posterior updates by adding these counts to Dirichlet parameters. For Gaussian data with unknown variance, the gamma distribution serves as the conjugate prior for precision (inverse variance). The Student-t distribution emerges when inferring both Gaussian mean and precision simultaneously, obtained by marginalizing out precision from the joint posterior, providing robustness to outliers through heavier tails. The exponential family framework is characterized by P(x|w) = h(x) × exp(w · T(x) - A(w)), where h(x) is the base measure, T(x) is the vector of sufficient statistics, w is the natural parameter vector, and A(w) is the log-partition function. This general form encompasses Bernoulli, Beta, Binomial, Dirichlet, Gaussian, Poisson, and other distributions. For any exponential family, there exists a conjugate prior P(w|α, ν) = g(w; α, ν) that results in a posterior of the same form, obtained by adding sufficient statistics to α and observations to ν.

Bayesian updating can be performed sequentially: after observing data (M successes, N-M failures), the posterior becomes Beta(a+M, b+N-M). This posterior then becomes the prior for the next observation. If we observe additional data (M' successes, N'-M' failures), the new posterior is Beta(a+M+M', b+N-M+N'-M'). This sequential updating is computationally efficient because we only need to add counts rather than perform full Bayesian integration.

Bayesian updating can be done sequentially: the posterior from one update becomes the prior for the next update. This allows continuous refinement of beliefs as more data becomes available. After observing a red ball then a blue ball, the posterior is proportional to i × (3-i), which is zero at i=0 and i=3, with maximum at i=1.5. This demonstrates that evidence can create uncertainty where none existed before, showing that Bayesian updating can resolve or create ambiguity about hypotheses.

Bayesian updates can be performed sequentially as new data becomes available. After each update, the posterior becomes the prior for the next update. The order of data processing does not matter, and you can either process data in small chunks or all at once—both approaches yield the same result. Each new piece of evidence shifts beliefs incrementally, with strong evidence potentially reversing previous conclusions. This sequential nature allows continuous belief updating as information accumulates.

When multiple pieces of evidence become available, Bayesian analysis allows sequential updating of beliefs. The posterior probability from one round of analysis becomes the prior for the next round. With each additional supporting evidence, the probability of the hypothesis increases incrementally. For instance, starting with a 25% prior belief, after one piece of supporting evidence it might increase to 37.5%, and after a second similar piece of evidence it might further increase to approximately 52%, demonstrating how accumulating evidence strengthens belief in a hypothesis.

Bayesian inference uses three key components: the prior probability P(B) represents initial beliefs before new data; the likelihood P(A|B) represents new evidence or update; and the posterior P(B|A) represents updated beliefs after incorporating evidence. The denominator P(A) normalizes the result. This framework enables sequential updating—using current posterior as prior for next experiment—forming the basis of Bayesian statistics. The coin flip example demonstrates how we start with a fair coin assumption (prior), observe data (flips), update our belief about fairness, and use this updated belief as prior for future experiments, creating a feedback loop of learning from data.