Academic Databases
Explore Our Semantic Databases
STEM & Tech
Math, Physics, CS & Engineering...
Economics
Macro, Micro, Econometrics...
Business
Management, Marketing, Finance...
Health & Life Sciences
Biology, Medicine, Neuroscience...
Humanities & Social Sciences
Psychology, Law, Philosophy, History...
General Studies
General & interdisciplinary studies...
Explore Topics
Features

Interpreting Black Box Models in Machine Learning: A Guide

Added: 2026-08-07

Deep Dive

Basic concepts of supervised machine learning, specifically classification algorithms and how models map input features to target labels.

Machine Learning Full course

Machine Learning Full course

100

Supervised learning focuses on learning a rule that predicts outputs accurately for new inputs. The model learns from labeled data where both inputs and correct outputs are provided. Key concepts include: (1) Dependent variable (target/outcome) - what we want to predict, (2) Independent variables (features/predictors) - inputs used to make predictions. Common algorithms include Linear Regression, Polynomial Classification, Logistic Regression, Support Vector Machines, Decision Trees, and Random Forest. Each algorithm has strengths for different types of problems. The goal is to learn a mapping from inputs to outputs that generalizes well to unseen data.

Click to watch →

Supervised Learning Algorithms Recap | Machine Learning Explained Simply

Supervised Learning Algorithms Recap | Machine Learning Explained Simply

197

Supervised learning is a machine learning approach where models learn from labeled data containing both input features and target outputs. The two main types are classification (predicting categorical/discrete outcomes like pass/fail, spam/not spam) and regression (predicting continuous/numerical values like house prices, exam scores). Classification algorithms include logistic regression, naive Bayes, decision trees, random forests, SVM, and KNN. Regression algorithms include linear regression, decision tree regressor, random forest regressor, and gradient boosting. The key difference is the output type: categorical for classification, continuous for regression. Both use input features but differ in target variable nature.

Click to watch →

Supervised Learning- Topic # 7- Urdu/Hindi

Supervised Learning- Topic # 7- Urdu/Hindi

2.1K

Supervised learning is a machine learning paradigm where models learn from labeled data containing both inputs and correct outputs. The model learns the mapping between inputs and outputs to make predictions on new data. Key concepts include labeled data, model training, and the distinction between classification (predicting discrete categories) and regression (predicting continuous values). Linear regression finds linear relationships between inputs and outputs. Classification algorithms learn decision boundaries that separate different classes. Support Vector Machine (SVM) finds optimal boundaries maximizing margins between classes. The kernel trick enables SVM to handle non-linearly separable data by mapping it to higher-dimensional spaces.

Click to watch →

Artificial Intelligence Techniques for Decision Making | Workshop

Artificial Intelligence Techniques for Decision Making | Workshop

212

Supervised Machine Learning requires input variables (X) and output variables (Y, also called labels or targets). Classification algorithms predict categories (e.g., spam vs ham email, cancer vs no cancer, customer churn vs no churn). Regression algorithms predict numerical values (e.g., sales forecast, house price, diamond price). The model learns from labeled examples and then predicts for new inputs.

Click to watch →

Machine Learning

Machine Learning

195

Supervised machine learning trains algorithms on data with known outcomes to predict unknown data. The 10 added algorithms include decision trees, forests, discriminant analysis, gradient boosted trees, linear models, naive Bayes, nearest neighbors, neural networks, and support vector machines. A wizard helps select optimal algorithms. Classification involves predicting categories (e.g., benign/malignant tumors) using features like radius, texture, perimeter, and fractal dimension. The target variable specifies what to predict, while features provide predictive information.

Click to watch →

Deep Dive

The distinction between interpretable 'white-box' models (such as decision trees) and complex 'black-box' models (such as deep neural networks or ensemble methods).

White Box Vs Black Box Models In Machine Learning- Data Science Interview Question

White Box Vs Black Box Models In Machine Learning- Data Science Interview Question

15K

In machine learning, white box models (like linear regression, logistic regression, and decision trees) have transparent, interpretable internal mechanisms where you can understand how features interact and how decisions are made, while black box models (like neural networks and ensemble methods such as random forest and XGBoost) have complex internal structures with many layers and calculations that make it difficult to visualize how weights change or how features interact, though black box models typically achieve higher accuracy for complex, non-linear problems.

Click to watch →

Kevin Lemagnen - Open the Black Box: an Introduction to Model Interpretability in Python

Kevin Lemagnen - Open the Black Box: an Introduction to Model Interpretability in Python

3.6K

White box models (like linear regression and decision trees) are easy to interpret because you can directly examine their internal structure and see exactly how each feature affects predictions. Black box models (like random forests, gradient boosting machines, and neural networks) are complex and opaque - you can't easily see how individual features contribute to predictions, making interpretation challenging.

Click to watch →

Boris Babic | The case against explainable artificial intelligence and machine learning

Boris Babic | The case against explainable artificial intelligence and machine learning

1.4K

A machine learning model is defined as a black box if the estimated function relating inputs to predictions cannot be understood at an ordinary human level. This typically occurs with complex models like deep neural networks that rely on many parameters and non-linear transformations. In contrast, a white box model is one where the estimated function can be understood at a reasonable human level, such as simple additive linear models with modest parameters or decision trees with few steps. The distinction matters because white box models provide actionable, interpretable information that users can work with, while black box models offer high predictive accuracy but lack transparency.

Click to watch →

[DMQA Open Seminar] Model-Agnostic Interpretation Methods

[DMQA Open Seminar] Model-Agnostic Interpretation Methods

5.6K

Interpretable models like linear regression and decision trees have built-in interpretation methods. Linear regression provides coefficients showing variable relationships. Decision trees allow visual tracing of decision paths. Ensemble methods like random forests and deep neural networks are inherently complex and considered 'blackbox' models. The key challenge is that interpretable models cannot simultaneously achieve high accuracy and interpretability, and different models use different interpretation methods that cannot be directly compared.

Click to watch →

Interpretable Machine Learning

Interpretable Machine Learning

496

Black box models are machine learning models so intricate that their internal workings cannot be easily understood or explained. The distinction between white box and black box models is primarily about complexity rather than model type. Even traditionally interpretable models like linear regression or decision trees can become black box models when they grow too complex—linear regression when it has many features, or decision trees when they grow beyond a certain depth. Key factors affecting interpretability include linearity (which makes models easier to understand), monotonicity (whether feature increases consistently affect outcomes), and feature interactions (how features relate to each other). Non-monotonic models are particularly complex because feature-outcome relationships are not consistent.

Click to watch →

Deep Dive

Fundamentals of Natural Language Processing (NLP) text representation techniques, such as TF-IDF or bag-of-words, which are typically used in spam detection.

TFIDF - Bag of Words Technique - DataMites Data Science Courses

TFIDF - Bag of Words Technique - DataMites Data Science Courses

9.5K

Text vectorization converts unstructured free text into structured numerical data for machine learning. The CountVectorizer extracts unique words and transforms messages into binary vectors, but frequently appearing words dominate the model. TF-IDF (Term Frequency-Inverse Document Frequency) addresses this by combining term frequency with inverse document frequency, nullifying common words across all documents. In spam detection, TF-IDF with Naive Bayes achieves 98% accuracy on 5,000 emails by valuing distinctive words while diminishing common ones. This enables effective text classification where meaningful patterns matter more than frequent but uninformative words.

Click to watch →

TF-IDF Vectorizer Python | Natural Language Processing with Python and NLTK

TF-IDF Vectorizer Python | Natural Language Processing with Python and NLTK

20K

TF-IDF (Term Frequency-Inverse Document Frequency) is a vectorization technique that creates a document-term matrix where each cell contains a weight calculated as TF × IDF, with TF measuring how frequently a word appears in a document relative to total words, and IDF measuring how rare a word is across the entire corpus; this weight indicates how important a word is for differentiating a specific document from others in the corpus, making it particularly useful for text classification tasks like spam detection.

Click to watch →

Text Representation | NLP Lecture 4 | Bag of Words | Tf-Idf | N-grams, Bi-grams and Uni-grams

Text Representation | NLP Lecture 4 | Bag of Words | Tf-Idf | N-grams, Bi-grams and Uni-grams

171.3K

TF-IDF (Term Frequency-Inverse Document Frequency) combines two metrics: term frequency (how often a word appears in a document) and inverse document frequency (how rare a word is across the corpus). Rare words receive higher weights because they better distinguish documents. The formula multiplies TF by IDF: TF = count(term in doc)/total terms in doc; IDF = log(total docs / docs containing term). This technique is widely used in search engines and information retrieval systems.

Click to watch →

Text Representation Using TF-IDF: NLP Tutorial For Beginners - S2 E6

Text Representation Using TF-IDF: NLP Tutorial For Beginners - S2 E6

92.5K

TF-IDF (Term Frequency-Inverse Document Frequency) is a text representation technique that addresses the limitation of Bag of Words by reducing the influence of generic words that appear frequently across all documents. It calculates two components: Term Frequency (TF) normalizes word counts by dividing by total tokens in the document, and Inverse Document Frequency (IDF) reduces the weight of terms that appear in many documents using the formula log(total_documents/num_documents_with_term). This ensures rare, meaningful terms receive higher scores while common generic terms get suppressed, making it effective for text classification tasks like categorizing news articles or e-commerce items.

Click to watch →

Entenda as técnicas Bag of Words e TF-IDF para Inteligência Aritifical. Lendo os principais papers!

Entenda as técnicas Bag of Words e TF-IDF para Inteligência Aritifical. Lendo os principais papers!

647

This comprehensive section covers the foundational concepts of text representation in Natural Language Processing. The presenter introduces the video series on reviewing academic NLP papers and explains that the core challenge in NLP is transforming text into numerical representations (embeddings) that machine learning algorithms can process. Key concepts include: 'document' as the basic text unit (reviews, tweets, book chapters), and 'corpus' as a collection of documents. The section then details Bag of Words (BoW), a technique that converts text into numerical vectors by counting word frequencies in a matrix format. The presenter explains sparse matrices for efficient storage and demonstrates how BoW loses semantic meaning and word order information. The section concludes with Term Frequency (TF) and Inverse Document Frequency (IDF), explaining how IDF uses the formula log(n/documents containing the word) to measure word rarity, with smoothing applied to prevent zero values.

Click to watch →

Deep Dive

Key classification evaluation metrics, including precision, recall, F1-score, and confusion matrices.

MÉTRICAS DE EVALUACIÓN MODELOS DE CLASIFICACIÓN SCIKIT LEARN | #35 Curso Machine Learning con Python

MÉTRICAS DE EVALUACIÓN MODELOS DE CLASIFICACIÓN SCIKIT LEARN | #35 Curso Machine Learning con Python

21.6K

This video explains how to implement key classification evaluation metrics using Scikit-Learn library, including confusion matrix (a 2x2 table for binary classification showing actual vs predicted values), accuracy (ratio of correct predictions to total predictions), precision (ratio of correct positive predictions to total positive predictions), recall/sensitivity (ratio of correct positive predictions to total actual positive instances), and F1 score (harmonic mean of precision and recall). Each metric requires actual test data and predicted values from the model for calculation.

Click to watch →

Precision , Recall , F1-score and Confusion Matrix | machine learning شرح عربي

Precision , Recall , F1-score and Confusion Matrix | machine learning شرح عربي

2.9K

In machine learning classification, the confusion matrix evaluates model performance through four outcomes: True Positive (correctly predicted positive), True Negative (correctly predicted negative), False Positive (incorrectly predicted positive), and False Negative (incorrectly predicted negative). Precision measures the accuracy of positive predictions (TP/(TP+FP)), indicating how many predicted positives are actually correct. Recall measures the model's ability to find all actual positives (TP/(TP+FN)), showing how many actual positives were correctly identified. F1-Score combines precision and recall using the harmonic mean formula (2×Precision×Recall/(Precision+Recall)), providing a balanced metric especially useful for imbalanced datasets where one class has significantly fewer samples than another.

Click to watch →

Machine Learning Full Course | Beginner to Advanced (FREE) 2026

Machine Learning Full Course | Beginner to Advanced (FREE) 2026

16.5K

A confusion matrix shows classification performance with four components: True Positive (correctly predicted positive), True Negative (correctly predicted negative), False Positive (incorrectly predicted positive), and False Negative (incorrectly predicted negative). Precision = TP/(TP+FP) measures how many predicted positives are correct. Recall = TP/(TP+FN) measures how many actual positives were correctly identified. F1 score is the harmonic mean of precision and recall, useful for imbalanced datasets.

Click to watch →

Class 10: Artificial Intelligence | One shot | Code 417 | Term 2 | Aakash Singh

Class 10: Artificial Intelligence | One shot | Code 417 | Term 2 | Aakash Singh

2.9K

Classification model performance is evaluated using a confusion matrix with four components: True Positive (correctly predicted positive), False Positive (incorrectly predicted positive), True Negative (correctly predicted negative), and False Negative (incorrectly predicted negative). Precision measures prediction accuracy: TP/(TP+FP). Recall measures coverage: TP/(TP+FN). The F1 Score combines both: F1 = 2*(Precision*Recall)/(Precision+Recall). Accuracy measures overall correctness: (TP+TN)/(TP+TN+FP+FN). Good models typically have F1 scores above 0.9.

Click to watch →

Confusion Matrix Explained Simply | Spam Filter Example + Accuracy, Precision, Recall & F1 Score

Confusion Matrix Explained Simply | Spam Filter Example + Accuracy, Precision, Recall & F1 Score

333

Four key evaluation metrics are derived from confusion matrix values: Accuracy measures correct predictions overall using (TP+TN)/Total Samples; Precision measures true positives among all positive predictions using TP/(TP+FP); Recall measures true positives among all actual positives using TP/(TP+FN); F1 Score is the harmonic mean combining precision and recall. Using example values (TP=90, FP=10, FN=50, TN=850), calculations demonstrate: Accuracy = 0.94, Precision = 0.9, Recall ≈ 0.64, F1 Score ≈ 0.75. Each metric answers different questions about model performance, with F1 providing a balanced view when both precision and recall matter.

Click to watch →

Interpretability

Why It Matters

Key Methods

Agnostic Tools

Shapley Values

Spam Use Case

Bias Fixing

Interpretability

2:09

Playing Section

1
Explains the need for interpretability as models grow complex.
2
Highlights the concept of a black box in modern AI systems.

Deep Dive

Advanced Explainable AI (XAI) frameworks, such as Integrated Gradients, counterfactual explanations, and Anchors.

Introduction to Explainable AI (ML Tech Talks)

Introduction to Explainable AI (ML Tech Talks)

61.2K

Explainable AI (XAI) is the research field focused on making machine learning model predictions understandable and trustworthy to humans. XAI encompasses two related but distinct concepts: interpretability (understanding how models work internally) and explainability (communicating model decisions in human-understandable terms). The field addresses the challenge that while complex models like deep neural networks achieve high accuracy, they often become 'black boxes' that are difficult to debug, audit, and trust. XAI methods help stakeholders—including engineers, model consumers, and regulators—understand model behavior, identify biases, ensure compliance, and build trust in AI systems. A key technique in XAI is Integrated Gradients, a post-hoc method that explains individual predictions by accumulating gradient information across evenly spaced steps between a baseline and input, providing feature attribution scores that reveal which input features drive model predictions. This technique addresses the limitation of standard gradients, which saturate for important features, by creating a more complete picture of feature importance through numerical integration.

Click to watch →

Introduction to Explainable AI (XAI) | Interpretable models, agnostic methods, counterfactuals

Introduction to Explainable AI (XAI) | Interpretable models, agnostic methods, counterfactuals

18.7K

Explainable AI (XAI), also known as interpretable machine learning (IML), is a field focused on making machine learning models transparent and understandable to humans. XAI employs two main approaches: intrinsically interpretable models (simple models like decision trees and linear regression whose parameters can be directly understood) and model agnostic methods (techniques like SHAP, LIME, PDPs, and ICE plots that can explain any blackbox model without requiring knowledge of its internal structure). Additional approaches include causal models, counterfactual explanations, adversarial examples, and model-specific methods. The field addresses the critical need for transparency in AI systems, especially when decisions affect people's lives such as mortgage approvals or medical diagnoses.

Click to watch →

Stanford Seminar - ML Explainability Part 3 I Post hoc Explanation Methods

Stanford Seminar - ML Explainability Part 3 I Post hoc Explanation Methods

This comprehensive section covers the foundational and advanced methods for explaining individual model predictions. It begins with post-hoc explanation principles, distinguishing between local and global explanations and their respective applications. The core technical content covers LIME (perturbing inputs and fitting linear models), SHAP (averaging marginal contributions across permutations), Anchors (rule-based local explanations), and gradient-based saliency maps (including SmoothGrad and Integrated Gradients improvements). The section culminates with counterfactual explanations and algorithmic recourse, addressing how to identify actionable changes to features that would flip model predictions. These methods are essential for applications like loan approvals where denied applicants need guidance on what to change, and they address growing regulatory demands for explainable AI in high-stakes domains.

Click to watch →

XAI 5주차 Input Attribution Methods 1 Integrated Gradients - Part1

XAI 5주차 Input Attribution Methods 1 Integrated Gradients - Part1

461

Integrated Gradients is an XAI method that satisfies three core axioms—implementation invariance, sensitivity, and completeness—by integrating gradients along a path from a baseline input to the original input, ensuring that attribution results are consistent across different network implementations, correctly identify variables affecting output, and provide meaningful explanations where the sum of all variable attributions equals the total output change.

Click to watch →

IADF School 2022: XAI for Earth Science

IADF School 2022: XAI for Earth Science

249

This section covers advanced XAI methods and frameworks for understanding model behavior. Grad-CAM (Gradient-weighted Class Activation Mapping) applies gradient-based principles to convolutional networks, computing gradients of target class scores with respect to layer activations and producing coarse localization maps highlighting important image regions. Causal explanations in XAI involve training separate models to learn explanations from data, defining causal loss based on prediction error changes when features are eliminated, and minimizing KL divergence between this distribution and model explanations. A recent classification framework categorizes XAI methods by three criteria: how features are eliminated, what is explained (model behavior, prediction changes, or loss), and how results are summarized. This framework helps researchers understand relationships between methods and identify appropriate approaches for specific use cases.

Click to watch →

Deep Dive

Using interpretability tools for model debugging, identifying feature leakage, and auditing models for algorithmic bias and fairness.

Responsible AI – Model Interpretability and Fairness | Microsoft

Responsible AI – Model Interpretability and Fairness | Microsoft

1.3K

InterpretML addresses the critical need for model interpretability across stakeholder levels. The toolkit serves three primary purposes: proving models make decisions based on meaningful signals rather than problematic patterns to non-technical stakeholders, auditing models to identify and improve weaknesses, and ensuring models avoid using sensitive attributes as decision factors. InterpretML supports both glassbox models (intrinsically interpretable) and blackbox explainers (applicable to any model type). The Explainable Boosting Machine (EBM) represents a breakthrough as a glassbox model offering accuracy comparable to LightGBM and XGBoost while maintaining complete transparency and capturing feature interactions. The interactive dashboard enables cohort analysis comparing model performance across demographic groups, identification of top important factors, and local explanation of individual predictions with perturbation analysis. Critical importance is placed on scoring/inference time interpretability, addressing GDPR's right to explanation requirement that affected parties must receive explanations for automated decisions impacting their lives.

Click to watch →

SHAP values for beginners | What they mean and their applications

SHAP values for beginners | What they mean and their applications

155.2K

SHAP provides multiple visualization tools for understanding model behavior: waterfall plots show feature contributions for individual predictions, force plots display prediction breakdowns, mean SHAP plots show average feature importance, swarm plots reveal SHAP value distributions, and dependence plots expose feature interactions. SHAP enables effective debugging by allowing close examination of incorrect predictions to identify problematic features. A key application involves identifying when models rely on spurious correlations—for example, an autonomous vehicle model that used background pixels for predictions failed when deployed in new locations with different backgrounds. SHAP helped diagnose this issue by revealing the model's unexpected feature dependencies.

Click to watch →

Responsible AI: Evaluating Machine Learning Models in Python

Responsible AI: Evaluating Machine Learning Models in Python

934

Model debugging involves three key areas: error analysis identifies where models make inaccurate predictions, revealing that high overall accuracy (e.g., 89%) may hide poor performance on specific demographics (e.g., 42% accuracy for certain groups). Data distribution analysis reveals overrepresented or underrepresented populations that cause model bias. Interpretability addresses the black box problem by understanding which features drive predictions, making models more like 'glass boxes' for transparency during audits. The demonstration uses a diabetic patient readmission prediction use case with the UCI Diabetic Data Set, showing how to split data, normalize, encode categorical variables, and create preprocessing pipelines before training classification models.

Click to watch →

Lecture 7: Interpretable Machine Learning

Lecture 7: Interpretable Machine Learning

395

Model interpretability serves multiple practical purposes: (1) Debugging models by identifying when models rely on wrong features (e.g., background patterns instead of actual objects); (2) Detecting bias by revealing whether protected attributes like race or gender influence decisions; (3) Providing algorithmic recourse by helping applicants understand why they were rejected and how to improve; (4) Building trust among stakeholders including doctors, regulators, and patients; (5) Enabling compliance with legal requirements like the EU's GDPR right to explanation. These applications demonstrate why interpretability is essential for responsible AI deployment in sensitive domains.

Click to watch →

Learn Live - Train a model and debug it with Responsible AI dashboard

Learn Live - Train a model and debug it with Responsible AI dashboard

968

The model debugging journey begins with error analysis, which reveals that while a model may have high overall accuracy, different data cohorts may have uneven error distribution. Some cohorts may have very high error rates while others have perfect accuracy. The performance evaluation dashboard allows users to analyze model performance by cohort, examining metrics like false positives, false negatives, and accuracy distributions. Fairness assessment examines whether models are making decisions differently across demographic groups and whether data is over or under-representing certain populations. Model interpretability serves two purposes: explaining decisions to users and enabling developers to understand what changes would alter outcomes. This transparency is critical for governance and accountability, allowing organizations to audit models and understand why decisions were made.

Click to watch →

Deep Dive

Deploying explanation models in production environments, including real-time explanation generation and MLOps integration.

Alejandro Saucedo: Guide towards algorithm explainability in machine learning | PyData London 2019

Alejandro Saucedo: Guide towards algorithm explainability in machine learning | PyData London 2019

5.9K

This section addresses deploying ethical ML practices in production environments. The explainer pattern enables deploying both models and explainers as separate services that communicate in real-time, representing a generic infrastructure architecture. Technical implementation involves packaging models with dependencies into Docker containers, defining computational graphs, and deploying to Kubernetes clusters. The process includes exporting model binaries, creating Python wrappers with predict functions, and exposing RESTful APIs. This infrastructure enables domain experts to review and validate model predictions continuously, creating opportunities for human-in-the-loop review while maintaining automated decision-making capabilities.

Click to watch →

W9_L5: Explaining machine learning models

W9_L5: Explaining machine learning models

533

Model explainability (XAI) is essential in MLOps for building trust, ensuring accountability, enabling informed decisions, and mitigating risks in AI systems; two primary techniques exist: SHAP (Shapley Additive Explanations), which uses game theory to calculate feature contributions by measuring average marginal contributions across all feature combinations, providing both local and global interpretability but being computationally expensive; and LIME (Local Interpretable Model-agnostic Explanations), which approximates blackbox models with linear regression around specific predictions by perturbing input data, offering fast local explanations but lacking global interpretability.

Click to watch →

Explaining Credit Decisions with Amazon SageMaker - Webinar

Explaining Credit Decisions with Amazon SageMaker - Webinar

2.2K

This section covers deploying explainable models in SageMaker for real-time inference. It explains: (1) Retrieving trained model artifacts from S3; (2) Defining SageMaker models with custom containers, entry points, and dependencies; (3) Deploying to dedicated instances creating endpoints; (4) The four-function endpoint architecture: model() loads assets and initializes SHAP explainers; input() processes incoming requests; predict() runs models and generates explanations based on requested entities; output() formats responses. The section demonstrates real-time prediction serving with integrated SHAP explanations, showing how to send sample data through the endpoint and receive predictions along with SHAP values and interaction effects. It illustrates dynamic experimentation by modifying feature values and observing changing explanations.

Click to watch →

How to Easily Deploy Your Hugging Face Models to Production - MLOps Live #20- With Hugging Face

How to Easily Deploy Your Hugging Face Models to Production - MLOps Live #20- With Hugging Face

6.5K

MLOps frameworks like Amazon ML Run provide four main components: feature stores for engineering, automated workflows for building models across distributed environments, real-time serving pipelines for multi-stage processing, and real-time monitoring for triggering retraining or alerts. Traditional ML operations require separate teams for code writing, containerization, and operations. MLOps automation eliminates these silos by allowing developers to write code in their favorite environment and automatically convert it into elastic, auto-scaling production-grade services. Real-time serving pipelines are multi-stage flows encompassing data transformation, enrichment, ensemble building, and post-processing, treatable as single entities for rolling upgrades. Automated model retraining workflows enable continuous improvement using fresh data, including data preparation, training, optimization (such as ONNX conversion), and deployment. The recommended iterative workflow involves starting with off-the-shelf pre-trained models, evaluating on business data, deploying if satisfactory, and retraining with fresh data to address concept drift.

Click to watch →

Use Visual Tools to Create Machine Learning Models with Azure ML Service

Use Visual Tools to Create Machine Learning Models with Azure ML Service

1.8K

This concluding section covers deploying trained models to production and maintaining them over time. Models are registered in a built-in registry with version tracking, tags, and hyperparameter documentation for reproducibility. Deployment options include real-time endpoints (API key-authenticated) and batch endpoints (bulk processing), leveraging Azure Kubernetes Service with Azure Container Registry for Docker images and scoring scripts implementing predict functions. Advanced monitoring detects data drift—when production data distributions differ from training data—triggering alerts for retraining consideration. The section also covers MLOps integration with DevOps tools for automated CI/CD pipelines, testing, and code submission automation. This complete operational pipeline enables organizations to move models from experimentation to production while maintaining ongoing performance oversight and implementing best practices for sustainable ML operations.

Click to watch →

Deep Dive

Understanding the mathematical foundations of game theory that underpin Shapley values used in SHAP.

The mathematics behind Shapley Values

The mathematics behind Shapley Values

59.3K

Shapley values satisfy four fairness axioms: (1) Efficiency—all value is distributed; (2) Symmetry—identical contributors get equal shares; (3) Null Player—zero contributors get nothing; (4) Additivity—combined games' contributions sum. Lloyd Shapley proved this is the only value satisfying all four. This mathematical foundation explains why Shapley values are considered fair—they systematically account for individual contributions and player interactions through these fundamental principles.

Click to watch →

05e Machine Learning: Shapley Value

05e Machine Learning: Shapley Value

18.6K

This section explains Shapley values through game theory examples. Marginal contribution is defined as payoff with versus without a player. Using two-player examples ($50k/$75k individual vs $125k together), the video demonstrates how averaging marginal contributions across all possible orders produces fair allocations. The methodology handles synergy (greater than sum of parts) and anti-synergy (less than sum) cases equally well, always satisfying the efficiency constraint. This foundational understanding translates directly to feature importance in machine learning.

Click to watch →

Understanding The Shapley Value

Understanding The Shapley Value

18.2K

The Shapley Value is a mathematical method for fairly distributing the total value generated by a coalition among its members, calculated by averaging each member's marginal contribution across all possible orders in which they could have joined the coalition, ensuring that the allocation satisfies three key axioms: symmetry (identical contributors receive identical allocations), dummy player (those who contribute nothing receive nothing), and additivity (allocations combine linearly across independent games).

Click to watch →

PyData Tel Aviv Meetup: SHAP Values for ML Explainability - Adi Watzman

PyData Tel Aviv Meetup: SHAP Values for ML Explainability - Adi Watzman

31.4K

This section derives SHAP values from cooperative game theory, originally developed by Lloyd Shapley in the 1950s. The core problem is fairly distributing a total payoff among players who contributed differently, with Shapley proving that only one unique solution satisfies fairness properties: additivity and consistency. The mathematical formulation defines SHAP values as the average marginal contribution of a feature across all possible feature subsets—calculating model outputs with and without each feature, then averaging differences. This theoretical foundation ensures SHAP provides the only mathematically unique solution for fair feature attribution. The section explains why this matters: without such a theoretical guarantee, different explanation methods could produce arbitrarily different results, undermining trust in model interpretability tools.

Click to watch →

Understand ANY Machine Learning Model

Understand ANY Machine Learning Model

32.1K

SHAP values are derived from game theory, specifically the Shapley value concept. The mathematical expression involves taking the model output for a sample (f(x)), subtracting the expected value of the model output (E[f(X)]), and then distributing this difference across all features. Each feature's SHAP value represents its contribution to moving the prediction away from the expected value. When aggregated across all features for a sample, these values sum to the difference between the actual prediction and the expected value.

Click to watch →

Interpretability

Why It Matters

Key Methods

Agnostic Tools

Shapley Values

Spam Use Case

Bias Fixing

Interpretability

2:09

Playing Section

1
Explains the need for interpretability as models grow complex.
2
Highlights the concept of a black box in modern AI systems.

687 views121:32:50@Liora_ioOriginal Release: 2023-04-29

Interpretability in machine learning refers to methods that help us understand how complex models make predictions, particularly important for black box models like deep neural networks. There are two main approaches: specific interpretability (examining model parameters directly for simpler models like linear regression) and agnostic interpretability (using approximation methods like LIME, SHAP values, or sensitivity analysis to understand complex models by examining input-output relationships). The key purposes of interpretability include ensuring models work as intended, detecting biases, meeting regulatory requirements, and improving decision-making by understanding which features drive predictions.

[00:00:09]hello everyone and welcome to this new video um I'm almond triet and I'm going to present today a small video about how to interpret a black box in machine learning so this video will explain to you some advice about how to use interpretability generalist picking it will be divided into two parts the the first one will be a small explanation of what is interoperability and what's the main uses of interpretability and Ai and the second part will be a use case about spam detection algorithm and we'll try to interpret how this algorithm is working okay so before we start I'll just check whether live is on yeah it seems like it's working um so this masterclass this uh presentation three uh will last for approximately one hour probably a little bit less um I will try to read your comments if you have any but as you can see uh I'm alone so it's going to be difficult for me to manage both their life and the questions so I won't answer any of the question uh as soon as you ask them but I'll try to check regularly if there are any um also this presentation will be recorded So if you are if you want to see it again or if you are seeing a ECR single already uh on YouTube you will always be able to see it again uh on our YouTube channel okay let's start so um the question of interpretability and modern machine learning is pretty recent uh of course machine learning is a new uh domain so it's hard to say that it has that any of the question about machine learning can be old but the the coefficient of how to make sure that the model is working as we wanted to do is a pretty new um the the thing is that uh 20 days ago it was very easy to understand how algorithms were working because they were fully coded by the informaticians the the the people that were creating the algorithms were controlling everything that was made by the machine or most of what the the machine was it was doing and if the machine was doing something that was not uh um uh made it to do the alpha magicians would have to correct it and it will not scare people nowadays if you ask the question to your neighbors or to your friends you will probably see that a lot of people are actually scared about what interpolations artificial and more generally speaking algorithm can do especially because we are aware of the fact that we don't understand everything inside those algorithms the thing is uh the machine learning and the intelligence artificial intelligence domain has considered durability rise during the last decade and uh nowadays it's uh very easy to create an algorithm that is so complex that you won't have enough of your entire life to understand it fully so the the question we want to ask today is how can you be sure that your model is doing what you wanted to do even if you don't understand everything that's happening inside it it's not an easy question there are many ways to do we'll try to uh give you uh an overview of the the different techniques that you can use and that you can Implement in your everyday lives okay so first of all let's talk about a concept that is used a lot in Intel artificial intelligence which is the concept of a black box usually a black box is something that we cannot look inside it's something that is uh made so that you won't be able to understand what's happening inside uh because you cannot access to it in machine learning or in artificial intelligence a black box is a little bit different because you could look inside you are able to open the black box and try to look at what's inside it but it's so complex that you won't be able to understand it even if you wanted to so what's happening is that you will usually have some inputs like a a video an image some text or I know the the the some information personal information about people the algorithm so that we will call Black Box here will make some transformation it will uh maybe make some operations with the the data you you gave it maybe he will create a random figures also he will use very complex uh neural network algorithm and then it will create an output which is the the prediction of the model so let's say you'd want to create a model that is able to predict uh the weight of someone based on his age his nationality and his uh size you will be able to have something here that makes a lot of operations in order to calculate the the the the weight of this person um the the main principle here to understand about interpretability is that you will have to look at both the inputs and the outputs to understand what's going inside the black box you most of the time won't look directly inside the Black Box because it's too hard it stays it takes too much time and it's easier just to look at what's the what's the relation between the input and the output the objective you want to uh get is to be able to in some ways predict what's going to be the results of your model when you use it on a specific input for example you want to be able to say that if you uh try your model if you execute your model in on a on a on a specific sample of someone that is as a specific size a specific age and a specific nationality you want to be able to say okay for this particular sample the model will predict this value all right I'll run this value um but what it is for why do you want to be able to predict the way the model is working there are many reasons why you would like it's to uh be that way the first one is that in order to convince maybe your boss or to convince legislators or your clients you want to be able to show to them that your model is working as they are expecting it to work so for example if you take uh the the Tesla car that is uh automated uh the Tesla car needed to be able to prove that even when it's working on a real street with real people or in front of it it won't have a strange Behavior so how did the the the the Tesla uh engineered uh worked in order to do so to convince people they created an interpretability model so they they tried to explain their black box which were the uh uh the current or the algorithm used in their cars and these black these interpretability model was able to show how the car was working especially uh how the algorithm that was given the camera uh uh the the the video of the car was working and if you try to look at it on the internet we'll be able to see that the the algorithm based on the video camera of the Tesla is detecting every single uh people and every single elements of the streets like uh the the roads the the the the lights the people the the other cars etc etc and they are showing so like little squares or one of those uh objects in order to not to to to work better but to to show to the rest of the world that it's working as we expect it to work so it's showing that it really detects uh people it really detects uh objects it really detects all the cars just like a human do and this is what we want him to do uh also in some specific cases you will need to show that your model meet legislative requirements let's take an example of a company that is building an algorithm in order to be able to classify the curriculum Vita is to say whether it's a a good or a bad curriculum beta is very simple um we would like to make sure that the model is not using uh information that they shouldn't use like for example the sex or the nationality or the age of the person we would like to uh classify and so interpretability model we make you able to make sure that the model is not using those forbidden information the third reason why we would like to use interpretability models is to detect potential biases so let's say you have a model that is working very well on the data you had in in hand you cannot be sure that this is going to work on the real data on the wheel in inside the Wheel World because most of the time the data you're giving to your model to train itself are very different to the data you're going to give the model in order to uh um to to make some real prediction the data you will have in your laboratory when you're training the model will most of the time be very very uh uh well liabilized there will be uh well organized there won't be so much errors and also it will be a little bit older than the real data so it will have probably a lot of biases let's take a a concrete example uh in a data scientist training we One Time created a project in order to detect uh kovid so the the the disease uh on uh lung radiography so we had a database of lots of lungs radios some of them were [Music] some of them had kovid some of them not and we wanted to have a classification model that was able to say whether this radio the the patient who who who who had this radio was uh given the kovid or not and the problem we faced is that very very quickly our model was able to classify very well those radio those radiograph radiography so we were a little bit doubtful and we used interpretability method to see where the model were looking at was looking at and we saw that the model didn't look uh too at the lungs like we expected it to do but we looked at everything that was outside the links like the small artifacts that were present on the radiography and those artifact didn't say anything about the covidian itself but it said a lot about the machine and the hospital where the radio was made and when we looked at it closer we saw that most of the radio that were classified as kovid were coming from the same hospital and inverse from the same machine and the same doctor so all those radio was at pretty much the same aspect with the same artifacts the same tools etc etc and the model was very strong to detect those uh those artifacts and not strong at all to detect covid so detecting potential biases is probably one of the most useful way uh to to the the the main reason why we want to use interpretability methods the fourth one is to help with decision so this is going to be very pretty quick if you have a model that is able to detect cancer for example a lung cancer you don't want him just to say okay this person this patient has cancer and this doesn't help because is very hard to trust 100 the model and you don't want to risk people's life in this decision so what you would learn what what you would like the model to do is to say okay this person has probably a cancer because I saw that there were a legend here in this radio for example and so the interportability method will give you not only the the the the the way the the model is working but also the the direction in where the model is looking at so for example I detected a cancer because of this specific bunch of pixels that is a little bit darker that's what it should be and so this will help the radiologist for example to finally make the final decision decision and which will not be only based on the model um for almost the same reasons you will be able to improve the personalities the performances of your model in some situation not all the time based on on those uh feedbacks so for example if you see that your model is not looking at the right direction is looking at the the top of the lungs while he should look at the the the middle of the lung you can maybe try to pre-process a little bit better your data uh because you know that if it looks at the middle of the lung it will have better performances finally you could also use these interpretability methods to understand better the problem so let's say you have a model again that is able to classify people to say whether they have cancer or not let's let's suppose that this model is having better results that even the best radial resist in the world you could try to see how the model is is working to try to understand why this is working better than the real geologist maybe because it's not only looking I don't know either the lesions in the lung but also the shape of the lungs or the size of the lungs and some things that weren't taken into account before and that you will be able to use in order to understand the problem better and to improve uh the the the way the the the classification is made okay so as you can see there are lots of reasons why we want to use uh interpretability um even if the the reasons don't apply in every single sector I believe that in every single uh project where there is artificial intelligence implied there is a a reason why we should use interpretability so now that we know what is interpretability and what's the reason why we need interpretability let's look at what's concretely interpretability so there are two main [Music] um way of using interpretability there are two main models of interpretability the first one is called Global interpretability the second one local interpretability for Global interpretability the objective is to explain the general Behavior and the Tendencies of the model so let's say you have your model that is able to make the classification between Cancer and Dot cancer the global interpretability will tell you that generally speaking your model looks at the shape of the lung or the size of the lung in in general for local and affordability it will explain the prediction of the model at the level of an individual a sample a patient so it will tell you that for this specific patient and not in general the model looked at the shape of the lung or it looked at the top of the line so of course this requires a greater Precision because you need to do it for every single individual or and we in the every single sample so it has a greater a smaller um uh effects on on the model um but this is uh in lots of cases what you want to have so for example let's have an another example another use case um if you as a bank are using a model in order to predict whether this is a good idea or not to give a loan to somebody if the model tells you okay no this is not a good idea to give a loan to this specific person you would like to be able to tell the person the reason why you weren't able to give them a loan and not only because the model said so probably you can tell it that you would be able to to give it uh to give him a loan if he came back in a few years because it's too young or maybe you will be able to give in a loan if he has a little big salary a little bit bigger or maybe if he's married I don't know so those features can be reached with local interpretability but if you only have a global interpretability method you will you you will just be able to say okay generally speaking if you are a little bit richer you will be able to have a loan but I'm not sure that in your specific situation it will works okay so we we saw that there were two main uh domains in interpretability but there are also two ways of uh to talk Talking of interpretability there are two main way of working uh the first one is what we called specific interpretability and I I'm personally think that this is not uh um the the this is not really interpretability when you think about it what I told you before with this slide is that when you are trying to look inside the black box you can not understand anything because it's too complex and so what you are going to do is that you won't look inside the black box but you will only look at the relation between the inputs and the outputs however in some specific cases that you may know for example when you're using a linear regression the black box is not very black it's like it's very easy to understand what's going inside and so you will be able to use specific interpretability so a model that is specific to linear regression to understand what's going on inside this blast box so even if I think that it's not interpretability properly speaking uh this is probably the easiest way and the better way to work when you are using something that is not as complex as a deep neural network for example so when you can when you have a linear regression logistic regression random Forest you should use specific interpretability to make sure that you have a strong compression of what's going inside the model um so what this means is that you will specifically look at the parameters that were calculated inside the black box and maybe compare them maybe add them together try to use some operation to give some feedbacks about how the model is working and for specific interpretability you can have both Global and local interpretability depending on the model you are using so uh specific antability of course is reserved for Less complex model you won't be able to use it for very complex model and the the very the thing that is very interesting is that there is no approximation necessary uh we will talk a little bit about it later but as you can see listening the relation between the input and the output uh is very vague like there are no only there is not only one way to do it there are lots of way to to look at it as it is and usually we we say that the interpretability methods will be approximations of what's going inside the black box it's like we will approximate how the inputs and the outputs are working together but for a specific interpretability this is not how it works sorry there is not any approximation used and so this is exactly how the model works um so if you want to use specific interpretability you would be able to use it with for example decision tree random Forest linear equation Logistics progression also with XG boost and to do so I'll try to show it okay um you will be able to do it using some methods some attributes based on the scikit-learn library so for example here you can see that there is an attribute called coeff and this attribute will give you the estimated coefficients for the linear regression problem if there are multiple targets uh linear and if there is one day of dance and features yeah so this will give you a way of understanding what are the coefficient that were learned in order to make the linear regression so for example if you have two inputs like X1 and X2 and one output is y linear regression will try to predict y with a linear combination of X1 and X2 so you will you will have two coefficient predicted let's say A and B so that you will try to optimize A and B so that a times X1 plus b times X2 will be close to y uh we can understand that A and B will give you all the information to need in order to understand how the model is working and for example we can say that if a is a lot bigger than b then uh probably X1 is a lot more important than X2 so they are there are some approximation about what I'm saying here but this is what you should remember of uh the specific interpretability of the linear regression uh I'll try to see if there are any questions before I continue yeah this is uh sorry you took comments okay hello to everyone who said hello uh let's continue feel free to ask a question if you if you had any time I won't answer right away but I'll I'll do my best um share screen okay so uh now that we know how specific interpretability works we'll talk about the big deal which is agnostic interpretability so as I said earlier what we would like to do is to understand what's inside the Black Box by looking at the Revelation between the inputs and the outputs so the agnostic method will use uh the the model it will like simulate try to simulate the model by giving some inputs and looking at the outputs there are many ways uh to to to to to use the agnostic methods but you need to keep in mind that this agnostic method is a model in itself it's like you use a model on top of a model this is why in some cases you will see the term middle model for the fact that you used a model to understand better another model um the the main reason why we will use agonistic interpretability is because we can use it to the whole category of models so any model that use inputs and output can be used with agnostic method however contrary to specific interpretability is based on approximations because you won't be able to get everything that is coming inside the black boss maybe there are some phenomena that you won't be able to see because you won't give the right inputs and you won't look at the right outputs uh let's take a concrete example if you look at the correlation between the inputs and the outputs so let's say we take the the example of the model that is trying to predict uh the the weight of someone based on his age his size and his nationality we will try to look at the correlation between age nationality size with the weight and it's uh probably that we will see that the the nationality is not related to uh so it's not correlated with the the the the weight but the the size is a lot correlated to the the weight so we will probably see some kind of uh collision and if the the the the the the the model is using those information you could write correctly we will be able to see that the the inputs like the the the size and the output the weight will be a lot correlated okay so this is a very very simple uh agnostic method there are others that are more complex that we will talk about it but you see the idea here we only need some inputs and some outputs we don't need anything else um okay so now that you see what is an interpretability model and that you see the difference between specific and agnostic we will try to get along some main groups of agnostic interpretability methods so the first one we'll talk about is the approximations so because we have very complex models here we would like to have something simpler here that will be able to do to do almost the same job so for example let's say we have a Transformers with convolutional neural networks and uh time series etc etc we would like to be able to do something that is similar to the black box but with a model that we will be able to interpret using specific interpretability uh to do so we will have many ways to do the first one is to say okay uh I will just use an um like a linear regression or a random Forest two in some ways simulate the black box but of course it seems a little bit weird because the reason why we used a black box is because we weren't able to make the right prediction with a simpler model like a random Forest however the agnostic method you need to remember is not based on the same data set it doesn't use the real label it only uses the output of the black box moreover let's say that your model needs lots of data but you don't have enough so the black box is based on a small amount of data your agnostic method will be able to have as many data as they want because they are not limited in the in that way uh the reason why is they can give some random inputs or to to to make uh even some inputs and look at what's outside so even if they are not real patients so let's say you you invent data about someone that is 10 years old and a French and uh that uh is one meter and 48 centimeters even if this data is not existing anywhere in the world you will be able to look at the output of the model and you will be able to use those in a couple of input and output to create the agnostic method to you to be able to to use this agnostic method so this is great because it means that you won't have any limit in terms of data it will be able to use your analytic method and to train it with as many data as you want but as you can see with the the example I took if you just create random inputs and you look at the output of the model you may have to look at some points in your space that don't exist and that won't ever exist like for example a 10 years old guy who who's with with a size of two meters this doesn't exist there is not any people in the world which with this uh specific information so what you will face is that this model this agnostic model will be based on some inputs that are not relevant some information that are not relevant so even if let's say you say okay uh I I can see that my model is uh giving a lot of importance to um I don't know the nationality which shouldn't be a something that it is important in that model uh maybe it's because you looked at some specific spaces in your data set that are natural events and your model so made irrelevant predictions about those data points so even if the agnostic method will give you a a strong uh and a very well good approximation of how your model Works generally speaking it won't give you a good approximation of how the model works in the space that you are looking at that is uh that you you you you want to give attention to so for example the the 10 year old people that have a size of uh one meter and 30 centimeters okay so the the problem you will have here that there is a the the there for for the agnostic method if you create new input you have no idea if those inputs are relevant or not so you have to be careful about that and so for this reason the the global approximation method that we we would like to use so uh to say okay we would like to do exactly the same thing as the black box but with a simpler model so for example like a random Forest you will you will you you will be able to do it your when the first model will work but the importances are the features of the the random Forest here are not the same at the one in the black box or at least or not the same in the uh in the right space for this reason when you use a global approximation method it doesn't work very well so for example if you use scikit-learn and you train a linear model on the output and input of the model so those are the labels you will prevent and those are the the the the input it won't work very well however there are other methods to do it so for for a local interpretation what you could say is that okay I I know that for example here this uh data set here is a data set with blue and red points and you can see that you you have access to only a few uh data points there uh so the the the the the the cross are uh let's say has a label equals to one and the the dots has has a label equals to zero this is just a classification a binary classification problem and the model that you trained the black box that you trained is represented here so you have in red all the points that's the model classified as red and in blue everything that the model classified as blue so as you can see the model has pretty good result because it has 100 accuracy but it's very hard to interpret you don't know how to interpret it so what you would like to use a simpler model to approximate how the the model is working but as you can see if you try to see to use a linear model you won't be able to have something as complex uh the model you have so what you're going to do is to approximate so you you still are going to approximate the model with a linear classifier but you will approximate it locally so let's say you select this cross here this Red Cross and you want to understand how the model is working next to this Red Cross so what you're going to do is to uh give to the model some a data bonds so those one two three four Red Cross and those blue points here those one two three four blue points and you will try to [Music] fit the prediction of the model on those little points near the the the the Big Red Cross here so how it works is that you won't look at all the inputs and all the outputs you have in your data set you will look at one specific input so you're we are in the local interpretability context and for this input you will look at all the input that are close to this one and you will train a very simple model so here a linear regression model a linear classifier model to um to to to make the prediction just like your model did um if we want to be more precise in reality it's working a little bit differently is that what I I telling you you won't just take into account those few points you will take all of the point into a coin which you will give more weight to the points that are closer to the Red Cross to the the sample you want to predict so for example this Red Cross here has almost no increase into the prediction of this model but those points here will have some and you can see that the the bigger the symbols or the bigger the crosses or the bigger the points the bigger the importance in the the prediction of the model so this is called a lime model so you have a library here that you can have a look on um you have the the the give them this is where I found the the the the image here I showed you and on on this GitHub page they will give you some example of use of the time algorithm uh okay so lime is for local interpretable modal agnostic explanations okay so these were approximations methods but there are other methods to try to understand models that won't use the simulation of the model so this is like kind of a simulation of the model and we can see that locally so close to this Red Cross it's not very far away from the real uh from the real uh classification so the real classification here is the Border here we can see between the red and the blue and the approximation is the line there so it's not exactly the same but it's close but for this um for this border it's not close at all like there is no no relation between this and this for example so it doesn't work on the whole data set again it only works locally so the second group of interpretability methods that we will talk about today are sensitivity analysis so sensitivity analysis is a big category there are lots of different models that talks about it I will talk mainly about two of them the first one is called swap importance it's from scikit-learn you can also see here Library if you want to you will find it just by typing swap importance um the the way it works is is that we say okay let's say we have those features so price bedrooms longitude latitude bathrooms it's like um the feature for an apartment and we would like to predict whether uh these uh let's say the the number of square meters inside this apartment um we would would like to predict the importance of each single feature inside this model the the principle we will use is that if a feature is not important so for example let's say the number of bathrooms doesn't have any uh importance in the prediction of the model uh we will see that if we take instead of the real value of the bathrooms the memorable bathroom will take random values we will not see any change in the prediction of the model so here we have a bunch of inputs and we will select one let's say the number of bathrooms and we'll say okay for the number of bathrooms I will take a random value instead of having the the the the correct values I will take random values and we'll I will look at what's the influence on the output if it has a big influence then we will see that this value is important if it doesn't have any influence then this feature is not important at all so here is a representation of what we could expect so for example for the price we will see that it has a big influence called the price is highly correlated to the the the number of square meters the same thing for number of bedrooms something for uh or not a lot for longitude and latitude and almost nothing for bathrooms um so these work pretty well but we will face uh some kind of problem the same problem we had before with the the approximations the global approximation is that let's say I I I I take a random number of bathrooms and I take like 10 bathrooms I say okay I I will I would like to to to to to see what's going on if I take 10 bathrooms randomly the problem is that the model is not made to make prediction on apartments with 10 bathrooms so the influence that will have this uh random function on the output is not predict predictable it's maybe it will have a high influence maybe it will have a very low inference but it's not something that would like to to to to know about this is not relevant um and so the the the the second problem we will face is the the the the the problem of the the correlation between the features let's say you have a feature that hit the price and another feature that is uh I don't know the other um the the so you have uh the the the the price of the the the apartment that you will pay and the price that the uh the the the people that are selling the apartment will receive uh there is a small amount of there is more difference between those two values but we can easily see that those two values are highly correlated because if you pay a lot of money then the the seller will receive a lot of money and reciproc and this is true in the in both ways um if you select one feature that you take randomly you won't be able to say okay I will take a random uh very small value for the price that I I receive and a very high for the price you receive this won't make any sense and the the prediction of the model of course will won't be relevant either so this are the problem with a scikit-learn swap importance and the the the way uh the the the the the the next feature the next uh um um the library is working is trying to to solve this issue by not taking just random values but by uh calculating the derivative of the value so instead of saying okay I take a random price it will say Okay I I will just lower a little bit the price or increase a little bit the price and see what the influence on the outputs so it will calculate the partial derivative of the of the of the price uh according to the the partial derivative sorry of the prediction according to the price and this is better because you won't have such big uh nonsense uh coming on on your on your inputs okay uh I'll try to see there are questions can you please speak a specific example how interpreter model not with a and b but in realistic case um I will give a specific example of how interpreter model uh in the second part of this presentation um if you if you have a specific example in in your head maybe you can try to to to to to to give it to me and I will try to to give you some answers um interpretability models are a very easy to use the main difficulties is how you understand how it works so the the [Music] the realistic problem will will try to to solve in the next part of the master class is how to interpret a Spam detector and how to understand the interpretability models of the spam detectors I I hope this will answer your question okay uh finally the the next category of um feature of interpretability is the the one based on the shareplay values so chaplain is a famous mathematician of the 20th century and he developed um a way to calculate uh how much money um um company should give to the people inside that company so it's based on Game Theory mathematics and it uses some axioms that are based on meritocracy and so I won't have time to give you a lot of information about this sharply algorithm but the thing is that the shop if I use have a very good very good properties so for example uh the chaplain values guarantees that if you have two uh people that are working exactly in the same way in the company they will receive the same amount of money for example um so if even if it's not a good idea to use sharply values in a real company in a real life because those um the the they are not aware of a lot of different factors we can use it in a lot of other domains and especially in interpretability so you will maybe ask me but what the the relation between sharply values and companies with the the machine learning in fact if if we try to think about it we could say that the the the company is just a bunch of people and they are working together to make uh some Revenue and for a um for machine learnings or in deep learning um it's just like a bunch of features that are working together or having some relation together in order to make a prediction and if we take this uh kind of a link between companies Chapel values and machine learning we can use the the the the definition of a shopping values and apply it to our algorithms so to be clear sharply values is just an equation that is giving you how much you need to give to to to to give how much money you need to give to to someone how or how much we evaluate the contribution of someone in a company and we will use an approximation of this because it's two uh time consuming to be able to use it in in a in AI so you won't use sharp evalues properly speaking you will use some approximation of it called sharp value and those sharp value will give you some feedback about okay so this specific feature uh contributed a lot to the prediction of the model and this other feature didn't contribute a lot in the prediction of the model uh there are there are a few libraries that can give you uh a nutrition of how sharp and sharper values are working and I will I will try to explain it to you in this specific example so um for example here we are going to use an uh a data set that or that is called eminist and it's just written figures so for example this is written figures and we trained an algorithm to try to predict what's the figure that is presented here so it's a classification algorithm and would like to say whether it's a zero a one a two Etc and here we calculated the sharp values of every single Pixel so pixels here are just like features in in tabular uh in in in trouble data sets and we would like to say what is the contribution of every single Pixel in the final prediction and before the the the prediction of the two for example uh for for the prediction of zero we can see that and there is no uh pixel that contributed to the final decision because everything is in uh white or in Gray so the sharp values of those pixels for the prediction of the zero is zero so for if if you don't really understand this is these are the pixels for the prediction of the zero pixel for production of the one pixels for breaks another two three four five six seven etc etc and here for example for the prediction of the two we can see that uh the the the the pixels on the top of the two and the pixel on the bottom of the two had a very high sharp values so those pixels had a very high contribution to the model so this is just like if you were saying okay here we can see that the model looked specifically at those um uh uh pixels here in order to make his prediction and of course this prediction was two that this number is two this figure is a two uh we can see the same thing for the one nothing for zero nothing for two nothing for uh five Etc But Here For example for number four we can see that there are some blue pixels here blue pixels is like a pixel that have a negative sharp values so this means that the model so that this uh this figure here looks a little bit like a four but she was expecting to see some some black pixels here he was expecting to see some information there to be able to say that this is a four and because it didn't see them it had a a negative contribution to the prediction so it's like I didn't predict for because I saw that there were no pixels no black pixels here and something for the nine we can see that I didn't predict nine because there are no black pixels here so it's only one uh stick and there is no curve here it's even clearer with the four so for example if you imagine that those two sticks are uh linked with each other it would be a nine but because it's not it's only a four here and so the the model saw that there were no any black pixels here so it it has a negative effect negative contribution to the prediction of the model but because there were no black pixels here the fact that they were white pixels on the top had a positive effect on the fact that the prediction was a four so you can see that even if there are white pixels the the model looked at it closely to make sure that this was a 4 and not a 9.

[00:58:25]okay so this is how sharp the values works it can be used on textual information it can be used on images it can be also used on tabularization there are a lot of way to use shop if you are interested in understanding how sharp works I strongly encourage you to look at this uh website here I will give you the link in the description of the YouTube video to to see how to use it and how how it works okay um before we start the use case I'll try to see there are any questions okay got any question for now let's go um so the use case will be about uh uh what you what to recommend in terms of data preparation to improve interpretability uh this this is uh uh a good question of course it depends a lot on the data you have initially um if your data set is images or textual information it will not be the same of course but to to interpret the the model the things you want to be to be uh to be sure that your data is is a quality it has a lot of not bad quality data because if you have bad quality data then your model will work uh like not randomly but it will take arbitrary decisions and the model the interpretability model will use those arbitrary decision to understand how the model is working which is not something you would like to have so you have to make sure the data has high quality and is big enough if you can of course so that you can see all the space where the model is making prediction because if too narrow then you will interpret only part of the model you will understand the model partially so you you will have to make sure that it it's a different Diversified enough um I I know that this is a very generic uh answer um I will try to maybe give you some ins in the the the their use case I will share with you so here uh I would like to uh use an a model that is able to detect spams so we have uh some emails that are coming in and my model will be able to say that some of the emails so the blue ones are not spams and some others the yellow ones or spams and afterwards we will use some interpretability method to try to understand what's going on inside this model uh oops here yeah um okay so I found a data set on the internet that was containing around 5000 emails that will build by humans so labeled by hand zero are the one that are not spam and one or the one that was spam they were collected in 2001 so they were a little bit old and from an American company that we will speak a little bit about afterwards and so let's look at it closely we will take an example here of a male that is not a Spam so we have some information about uh something uh coming on the next month and yeah on Thursday okay so because it's coming it's coming from a company it's mostly professional emails that are written in uh with a rich link wage but there are some exceptions inside and so the label for this specific for this specific mail is zero because it's leveled as not spam now let's see what's happening in spam labeled text we have some information about so it's a team of government officials coming from a committee and they will require your assistance in your mind that involved Investments of monies okay so they would like you to transfer money to their account and most importantly they would like to do it uh yeah immediately like a lot of times in spam they want you to give us money and immediately otherwise it will be too late so it's uh I didn't show you the whole span because it was very long but it's very uh like uh um obviously a Spam here and so the label is one uh okay so we will apply the usual processing step here so nothing very special we just split the data set into first half will be used for training second half will be used for testing if you are not aware of how this works it's just like we we have a whole data set that will cut in half to make sure that we will be able to test the model on something he never saw um now we will classically use the convectorizer method to have access to vectors instead of text this is usually something you would like to do at first to see if you can reach the goal that you want to reach in order to make your classification or the test you want to solve so convect eraser is just telling me that for example here in the first row of my data set so for my first email the word please appeared once the word tomorrow appeared once the word money appeared zero times and the word immediately appeared zero times and for the second row of my data set which is a spam here the word please and tomorrow appeared zero times but the word money appeared three times and immediately once so the the convectorizer coming from the library side of scikit-learn is very easy to use you just have to fit it on the train sets and then you can transform both the train and the test set so now I have a data set that is not containing any more uh words and texts but the number of occurrences of each word in each email and we'll uh as soon as it comes we will use a knife base model so from Sky could learn it's called multinomial and B and this knife base model will be able to make the classification between spams and not spams and when we so fit it on the train data set and test it on the test data set we see that we have an accuracy of 98 and on the the configuration Matrix we see that the results are very good so for example here around the true may also not the the the spams we were able to classify it correctly 734 out of the 729 emails so it's very very good and for the spams it's um uh very good also uh we were able to classify it correctly 285 emails out of the 296 spams so conclusion we are very happy because our model is a very very good to make classification and as soon as I see those uh scores I had to be a little bit suspicious because this model here is very simple in reality and it's hard to believe that with on the one small prediction I'm able to reach that level or performance of accuracy so before I go further we'll try to understand how our model works so um we'll try to look if it's working when I create my own email so here I created some random emails like click here to make one thousand dollars in one day and the prediction is still correct I see one here and hi can we meet this afternoon also this is probably not a Spam and the production of Barbados is that it's not a Spam so okay it I I I I tried to understand why my model can be biased or how my model was working but when I I test it it seems like it's working very well uh I I wasn't able to find any email that was giving me uh um oh sorry it was from the beginning I was wondering presentation uh sorry I came well we got a bit uh deformed you can see what I did um so this is my uh this is my data set I have the text here the label there I have access to some data there so you can try to look at it a little bit closer so it's classical males so these were for some um schedule times you know and here this is the spam so the team of government officials that request you immediately to contact to to be fully disposed and they would like to uh yeah to transfer to your account money so this is obviously a spam so here we just split the data set into and then we'll use the confectorizer which is going to give me some data set looking just like that so for example for each row I will have the number of occurrences of each word in my vocabulary so here please tomorrow monies for example for this row I know that the word please is here once the what tomorrow wants the word money zero times and for the spams it's pretty much working the same way and here I train my name base model there so I created a the multinomial knife base model here I fitted it on the X train with the label of a wide train and then I looked at the the the the results on the testing set so what I see is that using this very simple data set here so I only know which word appear in which uh email and I was able to see uh to to have a very very high accuracy so 98 of accuracy and if I look at the confusion matrix it's very very good so only 16 bad predictions over the whole data set over the whole testing data set and here I will just trying to make some prediction on my own emails so it's something that I wrote myself like click here to make one thousand dollars in one day or can we meet this afternoon so this is obviously a Spam and this is classified as a Spam and can we meet this afternoon is obviously not a Spam and is classified as not a spa so before we continue I will check if there are any question I'm sorry for this problematic um okay um the the the question now is to understand how we would be able to understand what's going on is that model we are of course a little bit suspicious because we see that this seems weird it's too easy uh it shouldn't be as easy as it is so we will have to use some interpretability methods to to make sure there is no uh no no no trick here used by our model so um knife base is a pretty simple model in fact we'll try to explain how it works quickly and then we'll try to interpret it concretely when we have those kind of uh information we can see that generally speaking over the 1000 spam that I have in my data set the word please appears 60 times and in my 2000 not spams email in my data set I have 600 emails with the word please so when I see the word please I have a strong opinion of the fact that it will probably be something that is not a Spam because here it's more likely to be something that is not a spam this is the theme principle with tomorrow where there are a lot more tomorrow words in the not spam category than in the spams category and so how how the the knife base model will work uh who will try to estimate the probability of an email to contain the word Liz knowing that it is a spam and the probability of a word to contain the word is knowing that it is not a spam so those two probabilities can be calculated that way so you just take the number of times the word please appear in the not spams category over the total of emails and same thing for the spaps so it's here it's 600 out of 2000 so it's 0.3 so 30 of the times and for the the spam it's six percent of the times only and so those two probabilities would be used in order to see whether this email is a spam or this email is not a Spam so how it's going to work we if we have an email let's say with only two words like please tomorrow uh this email the probability of this email to of being not a spam so probability of being not a Spam knowing that it contains those two words please tomorrow is proportional to the probability of uh containing please knowing that it's not a Spam times the probability of containing tomorrow knowing that it's not a Spam time the polity of being not a spam and which will do the same for the spam category um I I know that is probably a little bit hard to understand when I explained it just with my hands but quickly it's based on two things those figures here it's tags that they are more please in the notepam and the spams and the total number of spams here and it's it's uh take it by default it will consider that all the words are completely independent that we we won't use the dependence between two words in one sentence we'll say that they are just appearing randomly in this in the emails even if it's not really the case so it's a big approximated a big approximation here but it's still working pretty well um and so now the the question is how can I say whether so the the final classification how can I say whether the the male is a spam or is not a Spam I can just compare those two probabilities and I will say that the the the the the email belongs to the most probable uh uh category so for example for these I can see that if the the the the the the email is only containing the word please it's five times more uh likely to be not a Spam than being a Spam and for the world for the world uh money uh it's three times more probable more likely to be spam than not a spam and what you can see also is that those words here that we're looking at are also the features of my data set those are the features of my initial data set so I can use those features to say what is the importance of the the the the the the important that a model is giving to one specific feature so for example the feature please is very important because it's it's very easy to discriminate whether the the the email is a spam or not a Spam using the word please but let's say we have the word here which is a word very common that we will see in every email the pro the the ratio here will be close to one because it's not giving you any information about the fact that the the email with the spam or not the spam so this feature won't be very interesting as a contrary the the the word please and money are very interesting because they are giving you a lot of information so what you're going to do is to estimate the importance of the features based on those ratio so to be to be to be more specific we won't just look at those ratios but we'll use the maximum of the ratio and the inverse of the ratio to make sure that even if the the feature is close to zero it's it's still very interesting and so we will look not at 0.3 but 1 over 3 which is around 3.

[01:19:26]okay and so we will we will be able to to look at it directly on our data set so if we look at it directly in our data set um we can um yeah so I didn't show you how to get to this feature important there is a there is a few steps to go that I won't be able to explain to you but there are some calculations from a person to make first of all you take the maximum in the the between the the ratio or the inverse of the ratio then you take the logarithm of it to have something that is more coherent and then you just here I just printed the the the the 10 features with the maximum importance and here concretely I see that those features have some uh some names that I don't really understand here like who Enron ECT meter hpl they run etc etc and so there it seems like there is something wrong going with the importances of my features based on those importances here um okay I will make sure there is no question here yeah um I I voluntary didn't make distinction between the number of appearances and the the fact that the email is having the world there is a small approximation we are looking at the number of appearances uh so it's uh I when you see uh is 600 times the word please it means that when you make the sum of the total number of times the word please appear in the uh not spams it's reach 600.

[01:21:32]no okay so now that we are able to use uh interpretability method first of all let's try to see what kind of interpretability method you used um because it's um we used specific interpretability because we were able to because knife base is very simple we were able to look directly inside this algorithm we were able to look inside algorithm without using any agnostic method and also what I'm showing here is the importance of the features generally speaking so it's Global interpretability however if I would like to I would be able to see the importance of the features in one specific one specific email because I would be able to see if the the the word is appearing or not in the email and see what's their their the consequence of the the the the prediction okay so we stopped here so um I I can see that there are some strange words here but I would like to say whether so those words are informative for the model to say if it's a Spam or a not spam so as I told you before I said that for when it's uh bigger than one this ratio it will tell you that this is probably not a Spam and when it's lower than one it will tell you that it's probably a Spam so we will split this info importances in in two uh to see whether it's giving you the information of this is a spam or this is not a spam and when we do it we have those features so for the spam we have N bsp I don't really know what this means TD 2004 completion pills so pills let's like something they want to sell you and for the non-spam we have who and run ECT meter hpl so at this moment it seems like my model uh is biased because it gives a lot of importance towards that don't give ring me any bills so if I try to search on the internet for example for Enron what I will see yeah it's in French sorry um uh that Enron is a company uh based in Houston uh that was funded by etc etc and in the end of 2001 uh reported Financial condition was sustained by uh and so that the the the the the company went bankrupt because of Fraud and Corruption and so the if I look at it closely I will see that the the the the company was uh uh the the the the the legislation had to look at the emails of the company to say to to understand what was about these floated correction Scandal so in the end I understand that the the emails I was able to find are coming from this company here and one and the the the the the reason why I see that those words here like who uh Enron on ECT are so important is because it's very specific language uh for this uh company that is uh an energy company that use a lot of gas so for example if I try to look at the data I can see those words appear hpl uh who for Houston and run because it's the name of the company ECT because it's probably some kind of uh address that they use in the company meter this is something that they use to say that they are going to make some uh work on one specific place of the pipeline they use to to bring gas etc etc and those words are very very easy for the the on making the model very very easy to understand whether it's a Spam or not so for example if I take my uh example that I used before again like click here to make one thousand dollars in one day if I just add the name of the company and run then the prediction or is coming from not a Spam the spam to not a Spam and if I talk about some bills for example then the prediction will give me that it's a Spam because pills are categorized spams here for the conclusion I can make here is that even if my model is working very well on the the email I uh I used here 98 accuracy is very very good I won't be able to use it in a more generic way like I won't be able to commercialize this model because it's biased a lot so for example if I'm a writer of spams and I know that when I write and run this is not categorized as fund it is not detected then I will write and run in every single of my email and it will work very easily so in the end you can see that even if it's working very well there is probably still a lot of work to do to make sure that I will be able to get over those biases here uh shoot you to to to maybe try to use more emails coming from other other places and to anonymize maybe some of it uh to take off every single operation or appearance of the word Enron or who uh so it's probably going to be harder that what I I thought at the beginning okay so what can I do in order to correct those bias uh first of all the the the the the the the the main way to do it is to make some modification in the data set so maybe try to anonymize it or try to find more data but finding more data usually is very expensive because it needs to be globalized which can be very long or I can modify the functioning of the algorithm so for example if I just get out the the in my data set here if I take over the feature the column uh related to the Enron parts or the Houston part everything that has a link to the company I was trying to understand uh I will get rid of some biases that I have in my data set because it meets some human intervention it's usually very long and very complex so even if interpolivity is useful it won't give you the solution of the problem you will face so when you have a bias the intability model will tell you okay this is the bias you have but you won't have access to the the way to solve this bias okay uh I think this is it for my presentation I will stay a little bit to answer the question uh I hope you liked what you were able to see today uh there there were the the question of Anna about um the specific example I hope that the example I I I gave you at the end answered your your question otherwise I'm I'm still here for a few minutes to answer them thank you for your attention uh What What In this case using the external emails what do you mean by external emails um yeah so of course if we can have access to some other emails it will be better uh the the only thing is that we need to have a database of email that or that only have the labels otherwise it will cost a lot so we have to find somewhere uh a data set or we have to do it ourselves the data set with emails with the label spams or not spam oh okay you mean well okay an internally made within the company yeah probably you you [Music] I I don't know if the information can be found directly in the data set I found but yeah maybe based on the uh the the mail email address of the the sender we could be able to say whether it's a spinal or to simplify a lot the work of the people that will say whether it's a Spam or not foreign if there is not any more question we will end this session now thank you again for your attention this session will be available on YouTube uh and so if you are already watching on YouTube don't forget to like And subscribe if you like the video and to maybe watch other videos that we're making uh very regularly on this YouTube channel have a good day goodbye

Up Next

Machine Learning Explainability: Inherently Interpretable Models (Stanford Seminar)

@stanfordonline

19.3K views•2022-11-03

Building Real-Time ML Pipelines with Feature Stores and MLOps Frameworks

@ODSCAI

5.1K views•2022-02-20

Bypassing Tor Censorship: Bridges and Pluggable Transport Guide

@Coding_ForEveryone

397 views•2024-06-11

Neural Networks Explained: Math, Layers, and Learning Fundamentals

@3blue1brown

21.9M views•2017-10-05

Related Study Plans & Knowledge Roadmaps

Structured learning paths in Artificial Intelligence

View All Artificial Intelligence Topics

artificial-intelligence

Building Recommenders: Two-Tower & Vector DBs

01Foundations of Python & Machine Learning Basics

02Introduction to Recommender Systems

03Deep Learning & Representation Learning

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

Latent Diffusion: PyTorch, UNet & CLIP

01Deep Learning & PyTorch Foundations

02Convolutional Networks & Autoencoders

03The Math and Mechanics of Diffusion Models

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

NeRFs: 3D Reconstruction & Instant-NGP

01Math & Python Foundations for 3D Vision

02Photogrammetry & Camera Pose Estimation

03Neural Radiance Fields (NeRF) Core Concepts

+ 2 more learning modules

Explore Roadmap

artificial-intelligence

TFT Forecasting: Energy Demand & Deep Learning

01Programming & Data Science Foundations

02Introduction to Machine Learning & Time Series

03Deep Learning & Sequential Models

+ 3 more learning modules

Explore Roadmap