Explainable AI Explained: Introduction to XAI & Interpretable Models

Accumulated Local Effect (ALE) plots address the limitations of Partial Dependence Plots by integrating partial derivatives of the prediction function with respect to a feature of interest, which removes the main effects of other features and avoids the extrapolation issue while preventing omitted variable bias; the estimation procedure involves segmenting the feature range into intervals, computing finite differences at interval bounds, averaging local effects over the conditional distribution, and accumulating these effects to recover the global main effect, resulting in plots that are centered around zero and computationally more efficient than partial dependence plots.

Accumulated Local Effect Plots (ALEs) are a global model interpretation method that provides trend information similar to Partial Dependence Plots (PDPs) but with key advantages: they are robust to multicollinearity (highly correlated features) and avoid generating unrealistic feature combinations. ALEs work by dividing a feature's range into intervals, calculating the effect of changing the feature within each local interval by comparing predictions at the interval's minimum and maximum values, then accumulating these local effects across all intervals to produce the overall trend. The formal algorithm involves initializing the accumulated effect to zero, defining intervals, calculating local effects for each interval, accumulating the effects, combining feature values with accumulated effects, centering the values by subtracting the average accumulated effect, and finally plotting the result. ALEs can be implemented using Python libraries like Alibi Explain, and they provide clearer interpretations than PDPs when features are highly correlated, making them particularly valuable for understanding complex machine learning models.

Accumulated Local Effects (ALE) plots provide a visualization method for understanding predictor effects in black box supervised learning models by averaging local effects (finite differences) of predictions across the conditional distribution of other predictors, avoiding the extrapolation problems that plague partial dependence plots when predictors are correlated; ALE plots are computationally more efficient and produce more reliable interpretations of variable importance and interaction effects compared to existing methods.

This section covers two fundamental methods for understanding how machine learning models relate input features to outcomes. Partial Dependence Plots (PDP) visualize feature effects by averaging predictions while marginalizing other features, using Monte Carlo estimation since exact computation is infeasible. However, PDP critically assumes feature independence, producing biased estimates when features are correlated. Accumulated Local Effects (ALE) addresses this limitation by computing effects using conditional distributions rather than marginals, dividing features into intervals and accumulating local effects. Practical comparison on bike rental data demonstrates how ALE reveals more accurate patterns in extreme conditions where PDP fails due to correlation bias. Both methods enable global model interpretation without requiring model-specific knowledge.

Accumulated Local Effects (ALE) plots are a graphical technique for interpreting black box supervised learning models by plotting the predicted response versus individual predictors while accounting for correlations between variables; unlike partial dependence plots which require extrapolation and conflate predictor effects, ALE plots average the local effect (derivative) of the predicted response with respect to each predictor using the conditional distribution of other predictors, then integrate these effects to reveal how each predictor influences the response, including detecting interactions between predictors that would be missed by examining main effects alone.

This lecture covers methods for understanding what happens inside neural networks, including saliency maps, integrated gradients, occlusion sensitivity, class activation maps (CAM), and Grad-CAM for CNNs, as well as attention visualization, embedding analysis, and scaling laws for frontier models like transformers.

Class attribution map methods (CAM, Grad-CAM, Grad-CAM++) enable visualization of what CNNs focus on during predictions by using gradients and feature maps to generate class-discriminative saliency maps; CAM uses global average pooling and linear regression to weight feature maps for each class, Grad-CAM eliminates retraining by using existing gradients as weights, and Grad-CAM++ improves localization by applying pixel-wise weighting to address limitations with multiple objects and occlusions.

Integrated Gradients is a post-hoc method for explaining differentiable models by accumulating gradients across evenly spaced steps between baseline and input. Unlike standard gradients that saturate for important features, IG provides global importance scores. The mathematical foundation involves generating interpolated images via linear interpolation, calculating gradients between model outputs and each step, averaging gradients to approximate the prediction function, and scaling attributions relative to input. The trapezoidal Riemann sum method provides accurate convergence. IG attributions possess desirable mathematical properties including completeness, sensitivity, consistency, and linearity preservation. For visualization, absolute values of attributions across color channels are summed to return grayscale masks highlighting important pixels. Alternative visualizations preserve gradient sign direction across different channels to more accurately represent how features combine.

This section covers three interpretation methods for CNNs. Saliency maps measure the derivative of the CNN output with respect to each input variable, highlighting which input values most influence predictions. For CNNs with gridded inputs, this produces one saliency value per grid cell per channel (e.g., 4096 values for 32x32 grid with 4 channels). Unlike fully connected networks with scalar predictors, CNN saliency maps can be visualized as images on the same grid as input data. Class Activation Maps (CAM) highlight regions in the input that contribute most to a specific class prediction. Unlike saliency maps which show how to change predictions, CAM shows where the model found evidence for its actual prediction. Grad-CAM (gradient-weighted CAM) is more general than original CAM, which only works with global max pooling layers. CAM produces one value per grid point (non-negative), while saliency can be positive or negative. CAM is layer-specific, typically applied to the deepest convolutional layer. Backwards optimization creates synthetic inputs that either minimize or maximize the activation of a given neuron, producing adversarial examples. However, backwards optimization often produces unrealistic synthetic data (e.g., checkerboard patterns), requiring careful interpretation.

Grad-CAM (Gradient Class Activation Mapping) is a post-hoc explanation technique for convolutional neural networks that generates heatmaps to visualize which regions of an input image contribute most to a model's classification decision by computing gradients of the predicted class with respect to feature maps and applying weighted averaging followed by ReLU activation.

Regulations mandate AI explainability to prevent discrimination. In the US, laws prohibit discrimination based on protected attributes (age, sex, race, disability). GDPR Article 22 explicitly requires explanations for automated decision-making systems, known as the 'right to explanation.' Similar bills exist in several US states including Illinois, Washington, and Massachusetts, demanding algorithmic accountability. Explainability is needed throughout the entire AI lifecycle: during training to debug models; during QA to check compliance; after deployment for individual prediction explanations; during monitoring to ensure fairness, robustness, and unbiasedness; and feeding insights back to training for continuous improvement.

Under GDPR Articles 13, 14, and 15, individuals have rights to information about automated decision-making affecting them, with Article 22 providing specific requirements for solely automated decisions with legal or similarly significant effects. The ICO guidance identifies six key explanation types: rationale (how decisions were made), responsibility (who is accountable), data (what information was used), fairness (whether treatment was equitable), safety/performance (system reliability), and impact (consequences for individuals and society). Process-based explanations detail why particular systems were chosen, including development and deployment measures for fairness and safety. Outcome-based explanations allow individuals to understand specific decisions, data used, and challenge mechanisms. When human involvement assists AI output, explanations should clarify how and why human judgment was reached. Organizations should deliver explanations through appropriate methods (verbal, written, visual) considering reasonable expectations, accessibility needs, and whether explanations should be layered with essential information first. Procurement criteria should include explainability requirements, and the ICO is developing self-assessment tools to help organizations demonstrate compliance.

AI explainability is essential for clinician trust in AI predictions. While AI can achieve accuracy comparable to or exceeding human doctors, it requires well-labeled, expert-annotated training data to function reliably. Data bias detection through occlusion experiments reveals AI may rely on spurious correlations rather than clinically relevant features. Trustworthy AI encompasses four interconnected components: explainability, validity, responsibility, and privacy. Two major regulatory frameworks now mandate explainability: the EU GDPR (2016) grants individuals the right to explanation, and the EU AI Act (2024) stratifies AI systems with medical AI requiring mandatory explainability mechanisms. Effective explainability requires addressing four components: the object of explanation, available tools, the explanation product, and the target audience. Three explanation categories exist: local explanations for individual predictions, global explanations for model behavior, and data-level explanations for input-output relationships.

Ethical AI addresses three critical areas: (1) Bias detection - identify demographic imbalances and rebalance through resampling; (2) Fairness - detect when remaining features correlate with cultural discrimination even after removing explicit features, use synthetic data to offset discrimination; (3) Explainability - instrument models to generate heat maps showing feature contributions, calculate what feature changes would alter predictions. This transparency is essential for high-stakes decisions like lending.

Financial regulators expect explainability - the ability to justify AI decisions based on data analysis. This requires understanding what's happening, mapping data flow, and ensuring AI-informed human decisions. Ethical AI means being prepared to evidence decisions, not taking shortcuts, and not marketing stories disassociated from truth. Compliance with GDPR and regulations across UK, Europe, and US is essential. Barristers have stated certain AI practices break these laws. The truth requires people to be educated to ask the right questions.