Multiple Linear Regression Explained Simply | Statistics Tutorial

AIC (Akaike Information Criterion) and BIC (Bayesian Information Criterion) are penalized likelihood criteria used for selecting the best regression models by balancing goodness-of-fit with model complexity; AIC estimates the relative distance between the true likelihood and fitted likelihood, while BIC estimates the posterior probability of a model being true under Bayesian assumptions, with lower values indicating better models; other model selection methods include forward selection, backward selection, stepwise selection, all subsets regression, and Mallows' Cp criterion, each offering different approaches to identify optimal predictor subsets for a given dataset.

This section covers model selection strategies for choosing among competing regression models. Topics include: (1) Best subset selection, which evaluates all possible predictor combinations but becomes computationally infeasible as p increases; (2) Forward stepwise selection, which adds one predictor at a time based on greatest improvement in fit; (3) Backward stepwise selection, which removes predictors one at a time starting from the full model; (4) Model selection criteria including Cp (Mallows' Cp), AIC (equivalent to Cp for Gaussian GLMs), BIC (stronger penalty with log n factor), and adjusted R-squared, all of which penalize model complexity to prevent overfitting; (5) Cross-validation as an alternative approach to model selection that directly estimates prediction error.

The minimum adequate model retains only variables contributing meaningful explanatory power. AIC (Akaike Information Criterion) compares models by balancing goodness-of-fit against complexity—lower AIC indicates better models. Stepwise regression automates variable selection: backward elimination starts with all variables and removes non-significant ones; forward selection adds variables sequentially. Using the complete mtcars model with 11 variables, stepwise regression identified a three-variable minimum adequate model (weight, cylinders, transmission) explaining 84% of variation. This parsimonious model performs nearly as well as the full model while being simpler and easier to interpret.

AIC (Akaike Information Criterion), AICc (corrected AIC for small samples), and BIC (Bayesian Information Criterion) are model selection criteria that balance goodness-of-fit against model complexity by penalizing additional parameters; lower values indicate better models, with AICc being particularly useful for smaller datasets and BIC generally favoring simpler models more strongly than AIC.

When selecting between competing time series models (such as AR models of different orders), AIC (Akaike Information Criterion) and BIC (Bayesian Information Criterion) provide objective criteria for model comparison. AIC is calculated as 2k - 2L, where k is the number of parameters and L is the log likelihood, while BIC is calculated as ln(n)k - 2L, where n is the number of samples. Both criteria balance model fit (higher log likelihood indicates better fit) against model complexity (penalizing more parameters to prevent overfitting). The model with the lowest AIC or BIC value is selected as the optimal model, as demonstrated by comparing AR1, AR4, AR6, and AR10 models on catfish sales data, where AR6 was found to be the best fit.

Ridge and lasso regression are regularization techniques for handling multicollinearity and preventing overfitting. Ridge regression uses alpha = 0 and shrinks coefficients toward zero without setting them exactly to zero. Lasso regression uses alpha = 1 and can set some coefficients to exactly zero, performing automatic variable selection. Both methods require splitting data into training and test sets, defining a sequence of lambda values, and using cross-validation on the training set to select the optimal lambda. The best lambda is then applied to the test set to evaluate prediction performance (MSE). Ridge tends to perform better when many predictors are slightly correlated, while lasso is preferred when only a few predictors are expected to be important.

Overfitting occurs when a model performs well on training data but poorly on test data (low bias, high variance). Underfitting occurs when a model performs poorly on both training and test data (high bias, high variance). Ridge regression (L2 regularization) prevents overfitting by adding a penalty term λΣθⱼ² to the cost function, shrinking weights but keeping all features. Lasso regression (L1 regularization) can set weights to zero, performing automatic feature selection. Multicollinearity (highly correlated features) causes instability and is addressed by regularization. Practical implementation involves: importing libraries, loading data, splitting into X and Y, train-test split, standard scaling, model training, prediction, and evaluation using R² score. The choice between ridge and lasso depends on whether you want to keep all features (ridge) or select important features (lasso).

Lasso (Least Absolute Shrinkage and Selection Operator) regression uses L1 regularization to reduce overfitting by adding a penalty equal to the absolute value of coefficients. Its key feature is shrinking coefficients exactly to zero, performing automatic feature selection by removing irrelevant features. This is useful for high-dimensional datasets. Ridge regression uses L2 regularization, shrinking coefficients toward zero but rarely to exactly zero, making it suitable for handling multicollinearity (highly correlated features). Both techniques prevent overfitting but serve different purposes - Lasso for feature selection, Ridge for stabilizing models with correlated features.

Ridge and LASSO regression address multicollinearity by adding penalty terms to the sum of squared residuals: Ridge regression adds λ times the sum of squared coefficients (L2 penalty), while LASSO adds λ times the sum of absolute values of coefficients (L1 penalty). These methods shrink coefficients toward zero, ensuring unique solutions even when multicollinearity exists, though they prevent standard confidence intervals and hypothesis testing.

Regularization adds penalty terms to the loss function to prevent overfitting and improve generalization. Ridge regression (L2 regularization) adds the sum of squared coefficients, shrinking them toward zero without eliminating them. This stabilizes coefficients in the presence of multicollinearity. Lasso regression (L1 regularization) adds the sum of absolute coefficients, which can set coefficients exactly to zero, effectively performing variable selection by eliminating less important predictors.

Logistic regression is a GLM for binary (0/1) outcomes modeled using the Bernoulli distribution. The linear predictor remains the same as in linear models (sum of covariates multiplied by coefficients), but the link function is the logit function: log(μ/(1-μ)), where μ is the probability of success. This transforms the probability scale to the log-odds scale, allowing linear modeling of the transformed mean. The inverse transformation (logistic function) converts predictions back to probabilities: μ = e^η / (1 + e^η). This approach naturally handles binary data without requiring outcome transformations.

Generalized Linear Models (GLMs) provide a unified framework for statistical modeling through three key assumptions: (1) Y|X follows an exponential family distribution with natural parameter η, (2) the goal is to predict E[T(Y)|X], and (3) η = θᵀX. Applying this framework to binary classification with Bernoulli distribution yields logistic regression: h_θ(X) = 1/(1+exp(-θᵀX)). The sigmoid function emerges naturally from the relationship between the natural parameter and the mean. This demonstrates how GLMs unify seemingly different models—ordinary least squares and logistic regression become special cases of the same general framework, differing only in their choice of exponential family distribution.

This section provides a complete treatment of binary classification using logistic regression. The speaker explains why nominal data (categories without order like yes/no, bought/not bought) presents unique challenges: it lacks gradations and can only take values of 0 or 1. The solution involves predicting probabilities rather than actual values, which creates a new problem since probabilities cannot be negative or exceed 100%. Binomial regression (logistic regression) is introduced as the solution, based on the Bernoulli distribution for single trials or binomial distribution for multiple trials. The logit link function is thoroughly explained, showing how it transforms probabilities into log-odds through three steps: calculating odds (p/(1-p)) and then taking the natural logarithm. This transformation ensures the model produces valid probabilities while maintaining mathematical properties needed for regression analysis. The section demonstrates implementation in Python using the Titanic dataset, showing how to specify formulas, indicate categorical variables, and interpret coefficients in terms of log-odds. The speaker emphasizes that logistic regression produces an S-shaped curve that stays within valid probability bounds, unlike linear regression which can predict impossible values.

Generalized Linear Models (GLMs) are a flexible extension of linear regression that accommodate non-normal data by combining three components: a systematic component (predictor equation), a link function (which transforms predictions), and a random component (probability distribution of residuals). Unlike standard linear regression which assumes normal residuals, GLMs can handle various data types: logistic regression for binary outcomes (using logit link with binomial distribution), Poisson regression for count data (using log link with Poisson distribution), gamma regression for skewed continuous positive data (using inverse link), and ordered logistic regression for ordinal outcomes. A key distinction is that link functions transform only the prediction equation (not the residuals), unlike transformations which affect the entire dependent variable.

Logistic regression is a type of Generalized Linear Model (GLM) that extends linear regression by assuming the response variable follows a binomial distribution (Bernoulli case) and uses the logit link function to model the relationship between predictors and the probability of a binary outcome (0 or 1), where the log-odds of the outcome equals the linear predictor (log(π/(1-π)) = x^Tβ), allowing prediction of probabilities rather than just classification.