Explainable AI: LIME for Model Interpretability

Generalized Additive Models (GAMs) resolve the traditional tradeoff between model accuracy and explainability by training each feature independently using functions (often trees) and summing their contributions. This additive structure achieves accuracy comparable to complex tree ensembles while maintaining interpretability. The explainable boosting machine (EBM) implementation creates trees for each feature separately, then combines them through boosting. This approach allows researchers to visualize partial dependence plots showing how each feature affects predictions, enabling modification of model relationships based on domain knowledge. GAMs represent a powerful middle ground between purely accurate but opaque models and simpler but less accurate interpretable models.

Surrogate models train simple interpretable models (decision trees, regression) on complex model predictions to gain insights. Global surrogate models provide average understanding of complex behavior, while local surrogate models (LIME) zoom into specific regions where response functions approximate linearity. This combination enables both overall model comprehension and detailed local explanations for specific predictions.

This section explains how global surrogate models approximate black box predictions using simpler interpretable models. The naive approach treats black box outputs as ground truth labels and fits decision trees, but this fails because decision trees have limited representational power. Advanced regularization techniques address this: one uses neural networks to estimate surrogate complexity, while sparsity orthogonality regularization creates sparse neural networks with orthogonal weight vectors. This dual constraint ensures decision boundaries remain axis-aligned, enabling accurate approximation by decision trees. The steel plate damage classification demonstration shows how this approach produces short, human-readable decision rules that reveal model behavior.

Interpretable machine learning (explainable AI) addresses the need to understand why machine learning models make predictions, beyond just achieving good predictive accuracy. This field encompasses two main approaches: interpretable models (such as linear regression and decision trees that are inherently understandable) and model-agnostic explanation methods (like partial dependence plots, permutation importances, global surrogates, LIME, and SHAP) that can explain any black-box model after it has been trained. These techniques serve critical purposes including ensuring fairness, protecting privacy, verifying robustness, identifying causal relationships rather than mere correlations, and generating trust in AI systems. A key application is counterfactual explanations, which help users understand what changes they could make to alter an outcome—for example, helping a rejected loan applicant understand what actions might improve their chances.

Surrogate models create transparent proxy models (such as decision trees, random forests, or GLM) that approximate the behavior of opaque models. Instead of predicting the actual target variable, the surrogate model predicts the output of the original opaque model. High accuracy in the surrogate model indicates it successfully captures the opaque model's behavior, allowing analysts to examine which features drive predictions through the simpler surrogate model.

Empirical studies reveal fundamental limitations: gradient-based explanations remain unchanged when neural network parameters are randomized, failing to capture true model behavior. LIME explanations can be unstable with few perturbations, producing varied results across runs. More critically, post-hoc explanations can be adversarially manipulated through imperceptible input perturbations. Attackers create classifiers that appear unbiased on in-distribution data but rely on sensitive features like gender, while appearing to use legitimate features on out-of-distribution perturbations. Since methods like LIME/SHAP rely heavily on perturbations, they fail to detect such biases, allowing adversaries to hide model dependencies on sensitive attributes in regulated environments.

Post-hoc explanation methods like LIME and SHAP can be intentionally fooled by exploiting the fact that their perturbation-based sampling strategies generate data points that fall outside the true data distribution, allowing adversaries to train classifiers that behave differently on real data versus perturbed data, thereby masking discriminatory biases from explanation tools.

This talk demonstrates that post-hoc explanation methods like LIME and SHAP, which rely on input perturbations to interpret black-box machine learning models, are vulnerable to adversarial manipulation. Hima Lakkaraju presents a novel scaffolding technique that allows an adversarial entity to craft arbitrary, misleading explanations for biased classifiers without altering their actual biased predictions. The method enables the creation of classifiers that remain systematically biased—such as racially biased ones in COMPAS datasets—but produce explanations via LIME and SHAP that appear fair and innocuous. Through experiments on real-world data, she shows how these manipulated explanations can deceive domain experts into falsely believing the model is unbiased. The talk further includes user studies that reveal how such misleading explanations can distort human trust and decision-making in critical domains like criminal justice and healthcare. The findings highlight a significant risk in relying solely on popular explanation tools for auditing AI systems, as they can be easily fooled even when the underlying model is highly discriminatory. The work underscores the need for more robust evaluation frameworks for interpretability methods in high-stakes applications.

LIME's neighborhood width parameter (epsilon) is highly sensitive and significantly affects explanations—different values can produce different feature importance patterns for the same data point. LIME generates feature importance explanations showing how each feature affects predictions for specific points, with importance varying depending on the data point. For text classification, LIME represents text as binary vectors indicating word presence, creates neighborhoods by adding/removing words, and identifies key features like 'exclamation' for spam detection. For image classification, LIME uses superpixels (contiguous regions of similar pixels), perturbs these regions by changing colors, and identifies which regions most influence classification. Advantages include model-agnostic applicability, ease of use, and compatibility with various data types. Limitations include sensitivity to parameters, potential to ignore variable correlations, instability between runs, and potential for manipulation to hide biases. LIME should be used as one of several interpretability tools rather than a standalone solution.

Perturbation-based explanation methods like LIME and SHAP can be vulnerable to adversarial attacks where models appear fair on perturbed data but use sensitive features like race or gender on actual data, potentially misleading human users into trusting biased models; this occurs because these methods generate synthetic data points that differ from real-world distributions, allowing attackers to create explanations that hide discriminatory behavior while appearing reasonable to non-expert users.

Organizations are shifting from policy documents to 'policy as code,' embedding bias scans, fairness validations, and explainability checks directly into CI/CD workflows. When data scientists commit new model code, automated processes check for dataset imbalance, verify metadata completeness, and trigger ethical approval gates before deployment. Responsible AI sits within broader compliance frameworks like GDPR, HIPAA, and the EU AI Act, with alignment rather than reinvention as the goal. The EU AI Act popularizes risk-based classification: unacceptable risk systems inherently violate human rights (social scoring); high-risk systems include medical diagnosis, credit scoring, and hiring where outcomes deeply affect lives; limited-risk systems like customer service chatbots don't make critical decisions; minimal-risk includes spell checkers and personalization. Explainability answers 'why did it make that decision?' through feature importance heat maps, decision trees, and counterfactual examples. Transparency shares reasoning with stakeholders including regulators and internal reviewers.

AI models exist on a spectrum of interpretability: white box models (decision trees, linear regression) are fully interpretable, while black box models (deep neural networks with billions of parameters) are nearly impossible to understand. More powerful models that handle complex tasks tend to be less interpretable. Explanation techniques include: local explanations (SHAP values) explain why a model made a specific decision for one individual; global explanations (feature importance) show which variables matter most across the entire population; partial dependence analysis shows how predictions change as variables change; subpopulation analysis tests whether models perform equally well across different groups. Comprehensive bias analysis requires multiple complementary approaches: partial dependence analysis reveals whether models are biased against certain groups; proxy variable detection reveals indirect encoding of protected attributes; individual-level analysis examines how changing single variables affects specific predictions. Bias correction is an iterative process requiring collaboration between data scientists and domain experts: identify biases, discuss with experts to determine acceptability, add missing data fields or modify features, retrain the model, and repeat. The EU AI Act (expected 2023-2024) will require AI systems in high-risk applications to undergo rigorous validation including explainability analysis, data quality assessment, risk analysis, and ongoing monitoring plans. Organizations must maintain extensive documentation (80+ pages) to demonstrate compliance. The 'good enough' principle states that models should be deployed when sufficiently accurate and fair, not when they can be further improved. Excessive optimization (the 'cagoule syndrome') can introduce new biases and waste resources.

XAI (Explainable AI) refers to methods and models that make machine learning system behavior and predictions understandable to humans. XAI is essential for three main reasons: (1) Trust - users need to understand what models do and verify no bias exists, especially in applications like facial recognition; (2) Safety - explanations help debug systems and identify bugs that accuracy metrics miss, such as models focusing on irrelevant features like snow instead of actual objects; (3) Legal compliance - European regulations like GDPR require explanations for automated decisions, and the AI Act is further regulating ML usage. XAI libraries serve three contexts: experimentation (seamless tools), production (robust platforms), and research (open source contributions).

The EU AI Act establishes a risk-based classification framework for AI applications: (1) Level 0 - Applications with no risk to users, (2) Level 1 - Applications with limited risk requiring transparency measures, (3) Level 2 - High-risk applications that must comply with specific compliance requirements, and (4) Level 3 - Applications that are unacceptable and prohibited, such as certain biometric applications that could pose risks to citizens. This framework ensures that AI systems are regulated proportionally to their potential impact on individuals and society, driving the need for XAI solutions that can provide transparent, accountable AI systems.

The EU AI Act imposes significant compliance requirements: organizations must be able to explain and reproduce AI decisions, particularly for credit decisions; maintain data snapshots from model creation; and retain access to actual models used. Simpler models like tree-based models are easier to explain and justify, while models with billions of parameters create explainability challenges. Uncontrolled AI deployment across teams creates compliance risks because organizations cannot explain how models were produced. Common enterprise use cases include IT support automation using agentic AI with multiple collaborating agents, developer productivity improvements, full process automation of back-office operations, compliance analytics and KYC improvement, legacy modernization, and AI layering for unified analytics across multiple capital market systems.