Conjugate Priors Explained: Definitions, Examples & Why They Matter

Hierarchical models are Bayesian statistical frameworks where parameters themselves are treated as random variables with their own prior distributions called hyperpriors. Rather than arbitrarily choosing hyperparameters (like α=81, β=219 for a beta distribution), these values are drawn from hyperprior distributions. This creates a hierarchy: hyperhyperparameters determine hyperprior distributions, which determine hyperparameters, which determine the main model parameters. Each additional layer reduces the direct influence of arbitrary choices while allowing data to have greater impact on the final results.

Hierarchical models in Bayesian statistics treat parameters as random variables with their own distributions, creating a hierarchy where hyperparameters (like Alpha and Beta in a beta distribution) are themselves drawn from hyperprior distributions, allowing for more flexible and data-driven modeling compared to fixed prior parameters.

Hierarchical Bayesian models extend standard Bayesian inference by treating hyperparameters as random variables rather than fixed constants. The model structure is: X ~ f(x|θ), θ ~ g(θ|λ), and λ ~ h(λ). This hierarchical approach provides greater flexibility and control over prior distributions. A medical analogy illustrates this concept: doctors use family medical history (hyperparameter information) to make more informed treatment decisions for patients. Similarly, hierarchical priors allow Bayesian models to incorporate higher-level uncertainty structures, making them particularly useful when data is sparse or when relationships between parameters are complex.

In Bayesian hierarchical modeling, parameters that are shared across different groups or conditions are called hyperparameters, which themselves have prior distributions (hyperpriors). This approach allows information sharing across groups while maintaining parsimony and avoiding overfitting, as opposed to fitting separate models for each group. The choice of prior distributions for hyperparameters (such as using gamma distributions for standard deviation) can significantly impact the results, and researchers must balance model complexity with the need to capture underlying mechanisms rather than just fitting specific data.

This section covers the practical implementation of ARD algorithms and the broader Bayesian hierarchical modeling framework. The presentation explains that ARD uses iterative optimization to find maximum evidence hyperparameters, alternating between computing weight posteriors and optimizing hyperparameters using evidence functions. Since λ_i depends on γ_i and vice versa, no closed-form solution exists, requiring iterative methods like expectation-maximization. The scikit-learn implementation follows a structured workflow: initialize hyperparameters, compute weight posteriors, update hyperparameters, prune irrelevant features where precision exceeds a threshold, and check for convergence. The section also covers Bayesian hierarchical modeling, where hyperparameters (λ and α) are assigned gamma distribution hyperpriors, enabling automatic learning of optimal hyperparameter values from data. Predictions use the predictive distribution—a normal distribution centered on the posterior mean of weights multiplied by basis functions—providing probabilistic predictions with uncertainty estimates.

This opening section establishes the philosophical foundation for Bayesian analysis. The speaker introduces himself as a professor at UC Irvine who has contributed to statistics, epidemiology, and Bayesian biostatistics. He shares his personal journey from frequentist to Bayesian, influenced by Dennis Lindley's lectures at UC Davis. The core philosophy is that all priors are informative in some sense, and the term 'non-informative prior' is a misnomer. Even well-intentioned diffuse priors can be disinformative. The speaker emphasizes that priors should have physical meaning in the problems they address, and that scientists having differing priors is normal and essential to scientific progress. The goal is for posteriors to converge through appropriate data observation.

Non-informative priors aim to let data dominate inference by minimizing prior influence. The basic approach sets prior equal to one (uniform), making posterior proportional to likelihood. However, this depends critically on parameterization: uniform on θ (0-1) works, but uniform on odds (0-∞) or log odds (-∞-∞) becomes improper. Jeffrey resolved this by proposing priors invariant to parameterization, proportional to square root of Fisher information. For binomial models, this yields beta(1/2, 1/2) instead of uniform beta(1,1).

Non-informative priors aim to minimize prior influence, but this is harder than it seems. Uniform priors are not truly non-informative because they assign different probabilities to different orders of magnitude. Parameter transformation destroys neutrality—a flat prior on one parameter becomes concentrated on another. Jeffreys priors, proposed by Harold Jeffreys, are invariant under reparameterization using the Fisher information matrix. They are theoretically important but practically problematic: they can be improper, non-interpretable, and difficult for prior predictive simulations. Priors make sense only with the likelihood because their consequences are visible only through interaction with likelihood. The same prior can lead to completely different results with different likelihoods—a normal prior with Gaussian models behaves gently, but with Poisson models can create heavy tails. What matters is the support of the prior and whether it is compatible with the likelihood's support.

This section covers the foundational concepts of non-informative priors in Bayesian statistics. Non-informative priors allow practitioners to perform Bayesian inference without incorporating specific domain knowledge about parameters. For compact parameter sets, uniform priors are used; for infinite sets, improper priors proportional to 1 are employed. Jeffrey's prior provides a principled method for constructing non-informative priors based on the Fisher information matrix, being proportional to the square root of the determinant of this matrix. A key property of Jeffrey's prior is its invariance under reparameterization—when changing from parameter θ to φ(θ), the induced prior remains a valid Jeffrey's prior due to the Jacobian adjustment in the change of variables formula.

In Bayesian analysis, the prior distribution plays a vital role in drawing inference on parameters of interest. Priors can be subjective (based on personal belief) or objective (based on data or physical mechanisms). When nothing is known about a parameter, uniform priors are often used: pi(mu) = 1 for location parameters and pi(sigma) = 1/sigma for scale parameters. However, these are improper because they do not integrate to 1. Jeffreys proposed non-informative priors based on Fisher information: pi(theta) = [det(I(theta))]^(1/2). For binomial, this yields Beta(1/2, 1/2); for normal, pi(mu) = 1 and pi(sigma) = 1/sigma. A critical insight is that uniform priors are not invariant under non-linear transformations, as demonstrated by the log transformation example.

Thompson sampling is an acquisition function strategy that samples a function from the Gaussian process posterior and then selects the point that maximizes this sampled function. By drawing random functions from the posterior distribution, Thompson sampling naturally explores uncertain regions while exploiting promising ones. This approach has been successfully deployed at companies like Google for online advertising decisions. Different acquisition functions (UCB, EI, PI) perform differently depending on the specific problem, and combining multiple approaches can yield superior results.

The Two-Armed Bandit Problem illustrates the exploration-exploitation tradeoff: choosing between options with unknown reward probabilities. The Chili Monster demo shows a robot learning which eye to poke for cookies. Thompson Sampling uses beta distributions to estimate uncertainty about each option's true value. With few observations, distributions are broad (high uncertainty); with many, they narrow (high confidence). The algorithm samples from each distribution and picks the highest value, naturally balancing exploration and exploitation. This algorithm powers real-world systems like online advertising and recommendation engines.

This comprehensive section covers Thompson sampling implementation across different reward types. For Bernoulli rewards, beta distributions serve as conjugate priors/posteriors, with Bayesian updating incrementing α for heads and β for tails. The algorithm samples reward estimates from posteriors, computes empirical averages, and selects actions with highest sampled means. Critically, sample size K controls exploration: larger K produces exploitation-focused estimates, while smaller K encourages exploration through noisy samples. Remarkably, K=1 proves practical because peaked posteriors with sufficient data make single samples approximate true means well. The section demonstrates how this approach achieves logarithmic regret growth, matching optimal performance of UCB and epsilon-greedy while providing automatic exploration regulation.

Thompson Sampling is a Bayesian reinforcement learning algorithm for multi-armed bandit problems that balances exploration and exploitation by sampling action parameters from posterior distributions and selecting actions that maximize expected reward based on these samples, achieving near-optimal regret bounds.

This section presents Bayesian approaches to exploration, which maintain probability distributions over action values (posteriors) and update them as data is collected. Thompson sampling implements probability matching by sampling from posteriors and selecting the action with the highest sampled value. This automatically implements optimism in the face of uncertainty—actions with wider posterior distributions (higher uncertainty) are sampled more frequently because they have higher probability of being optimal. Unlike frequentist methods, Bayesian approaches naturally incorporate prior knowledge and adapt beliefs as more data is gathered. The value of information concept quantifies how much future reward an agent is willing to sacrifice to gain information about the environment, with information value depending on remaining budget and potential improvement from new information. These methods provide principled ways to balance exploration and exploitation based on uncertainty estimates.