Shapley Values in Machine Learning: Feature Attribution

This section provides technical details about LIME and introduces SHAP as a more rigorous alternative. LIME's process involves: selecting an instance, perturbing data to create nearby samples, weighting samples by proximity, training a simple model on weighted samples, and using coefficients as explanations. Advantages include model-agnosticism and human-understandable outputs. Limitations include computational expense from dense sampling, only providing local explanations, and variability from random sampling. The section then introduces SHAP (SHapley Additive exPlanations), based on Shapley values from cooperative game theory developed by Nobel laureate Lloyd Shapley. SHAP assigns each feature a contribution value by considering all possible combinations (coalitions) of features and evaluating marginal contributions. While the mathematics is complex, libraries handle implementation automatically. SHAP offers advantages: both local and global explanations, works particularly well with tree-based models, and is based on rigorous mathematical axioms.

This section compares different SHAP explainers and contrasts them with LIME. Kernel SHAP is model agnostic but slow even with approximation. Tree SHAP is 100 times faster but model-specific for tree-based models. Deep SHAP handles deep learning models, while Linear SHAP works for linear regression. LIME creates local surrogate models by sampling data around prediction points and training simple interpretable models. LIME is inherently local and cannot provide global explanations. LIME has consistency issues where different random seeds produce different explanations. SHAP provides more consistent results and supports global interpretation, making it more reliable for production use despite higher computational cost.

Two major explainable AI methods are SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations). SHAP, derived from Game Theory, treats features as players in a game where the prediction is the payout, calculating each feature's marginal contribution across all possible feature combinations. LIME creates local interpretable models to explain individual predictions. These methods provide different approaches to understanding model predictions, with SHAP being particularly useful for understanding feature contributions across all predictions in a unified framework.

Kernel SHAP approximates SHAP values by sampling feature subsets and fitting a linear regression model. The model inputs indicate whether each feature is present or absent, and the output is the model prediction. After training, the regression coefficients approximate the SHAP values. Unlike LIME, Kernel SHAP weights samples according to how much information they contain, giving more weight to subsets that reveal feature contributions.

Three fundamental methods explain model predictions across different data types. SHAP (Shapley Additive exPlanations) applies game theory to calculate each feature's contribution by averaging across all possible coalitions, using sampling techniques for estimation. Integrated Gradients addresses gradient saturation by integrating gradients along paths from baseline to input, though baseline choice critically affects results. LIME explains locally by fitting simple models to perturbed data points around instances of interest, weighted by proximity. Each method has strengths: SHAP provides global feature importance, Integrated Gradients handles pixel-level image attributions, and LIME offers local instance explanations. Understanding these core approaches enables practitioners to select appropriate tools for their specific explanatory challenges.

SHAP (SHapley Additive exPlanations) is a powerful Python package for understanding machine learning models, but it has four significant limitations: (1) Despite being theoretically model-agnostic, SHAP has not been implemented for all machine learning frameworks, making it less accessible for some users; (2) SHAP assumes feature independence during value calculation, which can lead to unrealistic predictions when features are correlated, as permuting values across their entire range may generate implausible combinations; (3) SHAP cannot be used for causal analysis—it only measures how important a feature is to a model's prediction, not how important that feature is to the actual target variable in reality; (4) Humans using SHAP plots are prone to confirmation bias and may create false narratives or force interpretations onto the data, similar to p-hacking. Users must remember that SHAP conclusions should never extend beyond the model itself.

When correlated variables are included in a model, SHAP values can show distorted interpretations of variable relationships. The demonstrated examples showed that when variables were correlated, SHAP values did not accurately reflect true relationships between variables and the target. When multiple highly correlated continuous variables are present, SHAP values tend to concentrate the importance on one variable rather than distributing it among all correlated variables. This differs from other feature importance methods like Random Forest or Gradient Boosting, which may distribute importance among correlated variables. This concentration can be both a strength (clear identification of important variables) and a limitation (may miss important information in other correlated variables).

Weighted SHAP values address the issue that some features may contribute to both good and bad predictions. The method assigns weights to SHAP values based on how well the model performs on the most important features, using the Area Under the Curve (AUC) metric. The weights are chosen to minimize the difference between predictions on the most important features and predictions on all features. SHAP values can be validated using the AUC metric by testing how well the model performs on the most important features. If the model performs similarly or better on the subset, the SHAP values are considered reliable. SHAP values have several limitations: they are only as good as the model they explain, they depend on the training data, and for tree-based models, SHAP values may be non-zero for features that the model did not actually use. When features are highly correlated, SHAP values may distribute importance among correlated features. SHAP can be applied to time series analysis by treating lagged values as separate features, and the additive property allows contributions of different lagged features to be summed to explain the total prediction.

SHAP has significant limitations: it cannot determine whether increasing or decreasing a feature would improve predictions, it assumes feature independence and cannot detect correlated features (which may split contributions between them), and it cannot explain features that are themselves not interpretable. Every feature must be defensible and understandable for SHAP to be useful. These limitations mean SHAP should be used alongside other analysis tools and not as a standalone explanation method.

SHAP values derive from Lloyd Shapley's 1950s game theory work on fair compensation distribution, satisfying efficiency and consistency criteria. The unique solution requires averaging contributions across all 2^N feature coalitions, which becomes computationally infeasible for large N. Approximation methods include sampling random coalitions and permutation importance. However, permutation-based approaches critically fail with correlated features because permuting one feature disrupts realistic relationships between dependent variables. For tree-based models, exact algorithms exist, but deep learning requires approximation. This limitation highlights that collinearity represents a fundamental data issue, not merely a model explanation challenge.

This video explains that as financial institutions adopt AI/ML, regulatory requirements mandate model explainability, requiring organizations to incorporate explainability, fairness, and interpretability from the initial model creation phase rather than treating them as afterthoughts. The presentation covers three types of ML systems (opaque, interpretable, and comprehensible), fairness-aware ML principles, regulatory frameworks like the Department of Defense's Ethical AI framework, and demonstrates how Amazon SageMaker Clarify enables model explainability and bias detection across the ML lifecycle through feature attribution using SHAP values and automated bias metrics for deployment monitoring.

Explainable AI (XAI) is a set of tools and frameworks that help us understand and interpret predictions made by machine learning models, addressing the critical need to understand why models produce specific results. XAI is essential for several reasons: it helps us understand model behavior, identify bias in datasets, detect data leakage, meet regulatory requirements in fields like healthcare and finance, and improve black box models by revealing which features impact predictions. XAI techniques are categorized into model-based approaches (interpretable by design, like decision trees) and post hoc approaches (applied after training, including LIME for local explanations, SHAP for feature contributions using game theory, and Partial Dependence Plots for understanding feature-target relationships).

SHAP (SHapley Additive exPlanations) is a unified framework for explaining machine learning model predictions by connecting game theory with local explanations, providing a theoretically grounded approach to feature attribution that satisfies fairness properties (local accuracy and consistency) while offering practical computational efficiency through polynomial-time algorithms for tree-based models, thereby addressing the critical trade-off between model accuracy and interpretability in high-stakes applications like medical decision-making.

AI models exist on a spectrum of interpretability: white box models (decision trees, linear regression) are fully interpretable, while black box models (deep neural networks with billions of parameters) are nearly impossible to understand. More powerful models that handle complex tasks tend to be less interpretable. Explanation techniques include: local explanations (SHAP values) explain why a model made a specific decision for one individual; global explanations (feature importance) show which variables matter most across the entire population; partial dependence analysis shows how predictions change as variables change; subpopulation analysis tests whether models perform equally well across different groups. Comprehensive bias analysis requires multiple complementary approaches: partial dependence analysis reveals whether models are biased against certain groups; proxy variable detection reveals indirect encoding of protected attributes; individual-level analysis examines how changing single variables affects specific predictions. Bias correction is an iterative process requiring collaboration between data scientists and domain experts: identify biases, discuss with experts to determine acceptability, add missing data fields or modify features, retrain the model, and repeat. The EU AI Act (expected 2023-2024) will require AI systems in high-risk applications to undergo rigorous validation including explainability analysis, data quality assessment, risk analysis, and ongoing monitoring plans. Organizations must maintain extensive documentation (80+ pages) to demonstrate compliance. The 'good enough' principle states that models should be deployed when sufficiently accurate and fair, not when they can be further improved. Excessive optimization (the 'cagoule syndrome') can introduce new biases and waste resources.

This section introduces Fairlearn, an open-source Python library for assessing and mitigating bias in machine learning models. The presenter explains that Fairlearn provides comprehensive tools including numerous fairness metrics and debiasing techniques, with a powerful MetricFrame extending pandas DataFrames for subgroup analysis. The section covers the four-fifths rule used by the US EEOC, requiring selection rates for protected groups to be at least 80% of the most favored group's rate. The presenter contrasts US regulation with UK/EU approaches, noting that applying US-style fairness auditing tools may violate UK discrimination law. The section demonstrates loading fairness audit results into the Equitas online tool, which provides a decision tree helping practitioners select appropriate fairness metrics based on whether their intervention is punitive or assistive and other contextual factors.