Beta Conjugate Prior to Binomial and Bernoulli Likelihoods

The posterior predictive distribution for binomial data with a beta prior and future sample size n* > 1 follows the beta-binomial distribution, given by P(y*|y) = [n*! / (y*! (n*-y*)!)] × [Γ(α*+β*) / (Γ(α*+y*) Γ(β*+n*-y*))], where α* = α + y and β* = n - y + β are the posterior parameters, and Γ is the gamma function.

The predictive distribution posterior is p(y_new|y) = ∫ p(y_new|theta) * p(theta|y) dtheta. For the Beta-Binomial model, this integral can be computed in closed form. For a single observation, p(y_new=1|y) = (alpha+y)/(alpha+beta+n). This is the probability that the next observation is a success, given observed data. The formula shows the predictive probability is a weighted average of the prior mean and sample mean, reflecting both prior beliefs and observed data.

When the prior is Beta(a,b) and the likelihood is Binomial(n,x), the posterior is Beta(a+x, b+n-x). This conjugacy simplifies Bayesian computation. The derivation uses the proportionality method: the posterior is proportional to θ^x(1-θ)^(n-x) × θ^(a-1)(1-θ)^(b-1) = θ^(a+x-1)(1-θ)^(b+n-x-1). For prior Beta(1,4) with data x=3, n=10, the posterior becomes Beta(4,11). The parameters update by adding successes to a and failures to b.

The predictive distribution for the Beta-Binomial model is derived using the law of total probability and Bayesian conjugacy, resulting in a Beta(a+x, b+n-x) distribution that combines prior parameters (a, b) with observed data (x successes in n trials), where the integral of the posterior times the binomial likelihood yields the predictive distribution proportional to θ^(a+x-1)(1-θ)^(b+n-x-1).

This video provides a comprehensive framework for understanding posterior predictive distributions in Bayesian statistics. Key topics include: (1) The dual purpose of posterior predictive distributions for prediction and model checking; (2) Why analytical computation is often intractable, necessitating sampling-based approximation; (3) The two sources of uncertainty represented: parameter uncertainty (posterior) and data-generating process uncertainty (sampling distribution); (4) The binomial model framework for binary outcomes with conjugate beta priors; (5) The sampling algorithm: iteratively sample θ from posterior, then x̃ from sampling distribution conditional on θ; (6) How increasing sample size affects the posterior versus the posterior predictive distribution; (7) The practical insight that sampling distribution variance often dominates overall uncertainty. This framework enables Bayesian practitioners to make predictions and validate models while properly accounting for all sources of uncertainty.

The Dirichlet distribution generalizes beta to multiple categories, serving as the conjugate prior for multinomial likelihoods. Defined as Dir(p; α₁,...,αₙ) = [Γ(α₀)/Γ(α₁)...Γ(αₙ)]∏pᵢ^(αᵢ-1), it models probability distributions over n categories. Expectations follow E[pᵢ] = αᵢ/α₀, where α₀ = ∑αⱼ. This framework enables Bayesian analysis of categorical data with tractable posteriors.

The multinomial distribution generalizes binomial to K outcomes, with probability proportional to ∏k qk^nk. Its conjugate prior is the Dirichlet distribution P(q|α) ∝ ∏k qk^(αk-1). Like the Beta distribution, the Dirichlet enables Bayesian inference through simple counting: posterior parameters are prior parameters plus observed counts. The normalization constant involves Gamma functions, demonstrating how exponential families enable tractable Bayesian updates across different distributional forms.

The Dirichlet distribution is the conjugate prior for the multinomial distribution, meaning their product yields another Dirichlet distribution with updated parameters α_i + m_i where m_i are observed counts. This conjugacy enables efficient Bayesian updating without complex integration. Parameters α_i can be interpreted as pseudo-counts representing prior beliefs. Visualization shows: uniform distribution when all α_i = 1; concentration toward center when α_i > 1; concentration toward boundaries when α_i < 1. These visualizations demonstrate how concentration parameters control the trade-off between exploration and exploitation in probabilistic models.

The Dirichlet distribution is the natural extension of the Beta distribution to multiple parameters, suitable for modeling probability distributions with more than two outcomes. It is the conjugate prior for the multinomial distribution, which generalizes the binomial to k outcomes. The Dirichlet has parameters α_1, α_2, ..., α_k and density proportional to ∏ θ_i^(α_i-1). When you observe multinomial data, the posterior is also Dirichlet with updated parameters α_i + counts_i. This conjugacy is essential for LDA, where both topic distributions over documents and word distributions over topics use Dirichlet priors.

The Dirichlet distribution generalizes the Beta distribution to multiclass settings with k possible outcomes. When observations fall into k categories with probabilities f_k, the Dirichlet prior is conjugate to the multinomial likelihood. The posterior parameters are the prior parameters plus the observed counts for each category. This allows tractable Bayesian inference over multiple probability parameters simultaneously, enabling uncertainty quantification for categorical data without resorting to numerical integration.

The beta-binomial model provides a complete Bayesian framework for binary outcome data. The binomial likelihood p(Y|θ) = C(n,Y)θ^Y(1-θ)^(n-Y) models counts of successes, while the beta prior p(θ) = θ^(α-1)(1-θ)^(β-1)/B(α,β) encodes prior beliefs about the success probability. Crucially, this pair forms a conjugate relationship: the posterior p(θ|Y) is also beta-distributed with updated hyperparameters α' = α+Y and β' = β+n-Y. This conjugacy enables analytical tractability, allowing direct computation of posterior summaries without complex integration.

This section defines the beta-binomial model and explains conjugacy. The beta-binomial model is a Bayesian model where: (1) The prior for π (0 to 1) is a beta distribution with hyperparameters α and β; (2) The likelihood follows a binomial distribution with n trials and probability π; (3) The posterior is also a beta distribution with updated hyperparameters α' = α + y and β' = β + n - y. This model is called 'beta-binomial' because it combines a beta prior with a binomial likelihood. The posterior belongs to the same family as the prior, which is a special property called conjugacy. A conjugate prior is a prior distribution that, when combined with the likelihood function, produces a posterior distribution that belongs to the same family as the prior. This conjugacy property allows analytical solutions without numerical integration or simulation. The general formula for the posterior in the beta-binomial model is: Posterior(π | data) = Beta(α + y, β + n - y), where y is the number of successes and n is the total number of trials.

In Bayesian statistics, the beta-binomial conjugate model provides a powerful framework for analyzing binary data where the likelihood follows a binomial distribution and the prior follows a beta distribution; this conjugacy ensures the posterior distribution also follows a beta distribution with updated hyperparameters (α + s, β + n - s), enabling straightforward computation of posterior estimates, credible intervals, and probabilistic inference about population proportions while incorporating prior knowledge through the choice of beta hyperparameters.

The Beta distribution is conjugate to the Binomial distribution, meaning that when you combine a Beta prior with Binomial likelihood, the posterior distribution is also a Beta distribution. This mathematical property allows for analytical solutions without requiring numerical methods. The posterior parameters are updated by adding observed successes to α and failures to β.

Building a beta-binomial model involves: (1) selecting beta hyperparameters reflecting prior beliefs, (2) collecting binary outcome data, (3) constructing the likelihood function, and (4) computing the posterior. The binomial distribution models binary outcomes (support/no support, obey/disobey). Critically, the beta-binomial model exhibits conjugacy—the posterior remains a beta distribution with updated hyperparameters. This mathematical property simplifies computation and maintains analytical tractability throughout the Bayesian workflow.

This section introduces the transition from analytical Bayesian methods to computational approaches. While simple conjugate cases (beta-binomial, gamma-Poisson) yield closed-form posteriors that can be summarized directly, real-world problems with 40-50 parameters require sampling methods. The fundamental challenge is that complex likelihoods multiplied by priors produce posteriors without analytical solutions, necessitating simulation-based approaches to approximate posterior distributions.

A conjugate prior is a prior distribution that, when multiplied by the likelihood function, produces a posterior distribution of the same family type. This mathematical convenience simplifies Bayesian calculations. Examples include: Normal likelihood with normal prior (self-conjugate); Binomial/Bernoulli likelihood with beta prior; Exponential/Poisson likelihood with gamma prior; Uniform likelihood with Pareto prior; Multinomial likelihood with Dirichlet prior. When choosing conjugate priors, the posterior retains the same functional form as the prior, making analytical solutions tractable without requiring complex numerical integration.

Conjugate priors are strong priors that limit the posterior to the same distribution class as the prior. This is problematic when actual prior beliefs cannot be expressed as conjugate priors, as the resulting prior may significantly differ from true beliefs. When conjugate priors are not used, Bayesian inference becomes analytically intractable because the model evidence (denominator in Bayes' theorem) requires computing a high-dimensional integral over all model parameters.

When analytical posterior solutions are intractable, Markov Chain Monte Carlo (MCMC) provides simulation-based approximation. MCMC iteratively proposes new parameter values and accepts/rejects them based on posterior ratios, eventually converging to represent the true posterior despite initial ignorance. Gibbs sampling handles conjugate cases directly, while Metropolis-Hastings uses asymmetric proposals with acceptance ratios. Proposal distribution width critically affects efficiency—too narrow causes slow mixing, too wide causes high rejection. Detailed balance ensures convergence to target distribution. This brute-force approach approximates complex distributions lacking closed-form solutions, enabling Bayesian analysis for models where analytical methods fail.

Conjugate priors simplify Bayesian computation because the posterior has the same functional form as the prior, allowing us to update parameters directly rather than performing complex numerical integration. For the Bernoulli/Binomial case, the Beta prior is conjugate to the Binomial likelihood, meaning the posterior is also a Beta distribution. This conjugacy property makes Bayesian inference tractable for many practical problems. Without conjugate priors, we would need to use numerical methods (like Markov Chain Monte Carlo) to compute the posterior, which is computationally intensive.