Academic Databases
Explore Our Semantic Databases
STEM & Tech
Math, Physics, CS & Engineering...
Economics
Macro, Micro, Econometrics...
Business
Management, Marketing, Finance...
Health & Life Sciences
Biology, Medicine, Neuroscience...
Humanities & Social Sciences
Psychology, Law, Philosophy, History...
General Studies
General & interdisciplinary studies...
Explore Topics
Features

Isolation Forest for Unsupervised Anomaly Detection | PyData London 2018

Added: 2026-08-07

Deep Dive

Fundamental concepts of unsupervised machine learning, including how it differs from supervised learning in scenarios without labeled training data.

Un-supervised Machine Learning

Un-supervised Machine Learning

1.6K

Unsupervised machine learning differs fundamentally from supervised learning in three key aspects: (1) No labels are provided in the data, (2) No supervisor guides the learning process, and (3) Only input features (X) are available without corresponding output labels (Y). The main type of unsupervised learning is clustering, which involves classifying data into different groups based on their features without prior knowledge of the categories. Examples include grouping people at an airport based on appearance, categorizing Umrah visitors by attire, or separating children from adults at a train station based on physical characteristics.

Click to watch →

Machine Learning (2025 Level II CFA® Exam – Quantitative Methods–Module 6)

Machine Learning (2025 Level II CFA® Exam – Quantitative Methods–Module 6)

26.6K

Unsupervised machine learning differs from supervised learning because it does not use labeled training data. Instead, it analyzes input data to discover underlying structures without corresponding target variables. Two primary applications include dimension reduction (reducing the number of input variables by identifying correlated ones) and clustering (grouping similar observations together). Unlike supervised learning which predicts known targets, unsupervised learning reveals hidden patterns within the data itself.

Click to watch →

APRENDIZAJE NO SUPERVISADO EN MACHINE LEARNING - Todo lo que debes saber

APRENDIZAJE NO SUPERVISADO EN MACHINE LEARNING - Todo lo que debes saber

19K

Unsupervised learning is a machine learning classification where algorithms process unlabeled data to discover hidden patterns and structures. Unlike supervised learning with labeled outcomes, unsupervised learning identifies underlying characteristics without predefined answers. It is divided into parametric approaches (assuming normal distribution with mean and standard deviation, using Gaussian mixture models and Expectation-Maximization) and non-parametric approaches (distribution-free, suitable for small samples). The core challenge is the absence of evaluation metrics since no correct answers exist to verify results. Clustering represents the most important unsupervised problem, organizing objects into groups where members are similar to each other and different from other clusters.

Click to watch →

8 - Fundamentos do Aprendizado de Máquina - Aprendizado Não Supervisionado

8 - Fundamentos do Aprendizado de Máquina - Aprendizado Não Supervisionado

113

Unsupervised learning differs fundamentally from supervised learning by focusing on pattern discovery rather than prediction. Unlike supervised learning which uses labeled data to train predictive models (classification or regression), unsupervised learning works with unlabeled data to uncover hidden structures. The knowledge emerges directly from the data itself, without target variables or predefined classes. This approach closely mirrors human learning, where we naturally observe and categorize the world without explicit instruction. Key advantages include eliminating the expensive and time-consuming data labeling process, enabling work with the vast majority of unlabeled real-world data, and reducing project costs. The probabilistic approach involves estimating data density to transform and extract knowledge from the input data.

Click to watch →

Machine Learning Interview Questions & Answers | Machine Learning Interview Preparation |Simplilearn

Machine Learning Interview Questions & Answers | Machine Learning Interview Preparation |Simplilearn

224.3K

The fundamental difference between supervised and unsupervised machine learning lies in the use of labeled data. Supervised learning requires labeled data for training, where each observation has a known correct answer. Unsupervised learning works with unlabeled data, where the model must discover patterns and structure on its own. The presence or absence of labeled data is the key distinguishing factor between these two main types of machine learning.

Click to watch →

Deep Dive

The basic structure and mechanics of Decision Trees and ensemble methods like Random Forests, as Isolation Forests adapt these partitioning concepts.

Anomaly detection with Isolation Forests

Anomaly detection with Isolation Forests

25.7K

While random forests build decision trees to partition data into homogeneous groups (where each partition contains similar instances), isolation forests build isolation trees with a fundamentally different objective: instead of creating homogeneous partitions, they create partitions such that each data point becomes isolated in its own separate partition.

Click to watch →

ML-2-Decision Trees and Random Forest (Lecture)

ML-2-Decision Trees and Random Forest (Lecture)

2.6K

Ensemble methods partition data into different training samples for each tree. Techniques like bootstrapping create multiple training samples by sampling with replacement. This partitioning increases model diversity and improves ensemble performance.

Click to watch →

Машинное обучение 6. Decision trees and Ensembles

Машинное обучение 6. Decision trees and Ensembles

10.8K

Decision trees are piecewise constant functions that recursively partition feature space using threshold-based splits, with entropy (Shannon's information theory measure) and Gini impurity serving as primary criteria for evaluating split quality; ensemble methods like bagging (bootstrap aggregating) and Random Forest combine multiple trees trained on bootstrap samples to reduce variance and improve generalization, with Random Forest further enhancing diversity by randomly selecting subsets of features at each split, while out-of-bag estimation provides an efficient validation mechanism without requiring separate test data.

Click to watch →

Основы машинного обучения, лекция 13 — решающие деревья, бэггинг, случайные леса

Основы машинного обучения, лекция 13 — решающие деревья, бэггинг, случайные леса

2.3K

Decision trees partition feature space hierarchically using splits. Entropy (H = -Σ p_i log(p_i)) and Gini impurity measure node purity. Weighted impurity accounts for sample distribution across children. Stopping criteria include maximum depth, minimum samples per leaf, and minimum impurity reduction. Exhaustive tree search is computationally infeasible, necessitating greedy algorithms that make locally optimal split choices. Tree construction evaluates splits by calculating weighted impurity for each possible threshold. Decision trees overfit training data by creating complex boundaries that memorize examples. Ensemble methods address this by combining multiple models trained on different data subsets. Bagging (Bootstrap Aggregating) trains models independently on bootstrap samples and combines predictions through majority voting or averaging. Ensemble methods work best when models are diverse—making different errors that cancel each other out.

Click to watch →

Random Forest Algorithm - Random Forest Explained | Random Forest in Machine Learning | Simplilearn

Random Forest Algorithm - Random Forest Explained | Random Forest in Machine Learning | Simplilearn

363.7K

Decision trees form the foundation of random forest, using recursive splitting based on entropy and information gain principles. Entropy measures data randomness, while information gain quantifies how much uncertainty is reduced by splitting on specific features. Trees consist of root nodes (initial splits), decision nodes (branching points), and leaf nodes (final classifications). Random forest extends this by creating multiple trees from randomly sampled data and features, then combining their predictions through majority voting. This ensemble approach reduces variance, handles missing data gracefully, and maintains high accuracy even when individual trees have limited information.

Click to watch →

Deep Dive

The core definition of anomaly detection (or outlier detection) and the unique challenges it poses, such as extreme class imbalance.

Anomaly Detection for Real-World Systems by Manojit Nandi | DataEngConf NY '16

Anomaly Detection for Real-World Systems by Manojit Nandi | DataEngConf NY '16

8.7K

An anomaly is something noticeably different from other things, though no rigid formal definition exists. Key challenges include distinguishing true anomalies from emerging trends (like Pokémon GO initially appearing anomalous but becoming normal behavior) and recognizing adversarial settings where attackers intentionally evade detection. Two primary approaches exist: statistical modeling (defining normal behavior distributions and identifying low-probability points) and machine learning classification (training models to distinguish normal from anomalous behavior). The latter faces class imbalance challenges since anomalies are rare events. Specialized algorithms have been developed to address these unique challenges in anomaly detection.

Click to watch →

[핵심 머신러닝] Anomaly Detection (개요, 확률분포기반)

[핵심 머신러닝] Anomaly Detection (개요, 확률분포기반)

7.4K

Anomaly detection deals with highly imbalanced datasets where normal data constitutes the majority (e.g., 99%) and anomalies a small minority (e.g., 1%). Applying supervised learning to such data creates models biased toward the majority class, resulting in high accuracy but poor anomaly detection. The solution is unsupervised learning, which uses only normal data to learn what constitutes 'normal' behavior. Anomaly detection is fundamentally a one-class classification problem where the model learns to recognize normal data and identifies anything outside this learned pattern as an anomaly. The model creates a boundary encompassing normal data, and points outside this boundary are flagged as anomalies.

Click to watch →

Applied AI Study Group Week 2/2 - Anomaly Detection

Applied AI Study Group Week 2/2 - Anomaly Detection

181

Anomaly detection is the process of identifying unusual patterns within systems or processes. It has critical applications across multiple domains: manufacturing (detecting abnormal machine sounds), healthcare (identifying diseases like cancer from medical imaging), and finance (detecting fraudulent activities). The fundamental challenge lies in class imbalance, where normal data vastly outnumbers anomalous data (e.g., 99% normal vs. 1% anomalous). This imbalance makes model training difficult, as models may learn to classify everything as normal, achieving high accuracy without actually detecting anomalies. Traditional evaluation metrics like accuracy become misleading in such scenarios, requiring alternative metrics like precision, recall, and F1-score for proper assessment.

Click to watch →

Anomaly Detection - Nick Radcliffe

Anomaly Detection - Nick Radcliffe

23.6K

Anomaly detection identifies data points deviating from expected patterns, with applications spanning fraud detection, system monitoring, medical diagnostics, and opportunity identification. Core challenges include: no absolute right answers without context—statistically unusual values may represent legitimate phenomena; extreme outliers dominate standard visualizations requiring specialized techniques like log-log plots; and distinguishing genuine anomalies from expected variations demands deep contextual understanding. Evaluation requires balancing four metrics: True Positives, False Positives, True Negatives, and False Negatives. A critical paradox emerges: even with 80% true positive rate and 10% false positive rate, only ~7.5% of flagged items may be genuine when defects are rare. Optimal threshold selection requires balancing trade-offs based on relative costs of false positives versus false negatives, with no universally correct solution.

Click to watch →

89. 모두가 잘 알고 있다고 생각하지만 사실 그렇지 않은 오토인코더 기반 이상치 감지에 대하여 (서울대 대학원 윤상웅)

89. 모두가 잘 알고 있다고 생각하지만 사실 그렇지 않은 오토인코더 기반 이상치 감지에 대하여 (서울대 대학원 윤상웅)

4.6K

Outlier detection is a binary classification problem distinguishing normal data (following regular patterns) from abnormal data (deviating in countless ways). The fundamental challenge is that there are infinitely many ways data can become abnormal, making it impossible to collect comprehensive training examples for all outlier types. Supervised learning approaches fail because they require labeled outlier data, which cannot be collected exhaustively. This motivates unsupervised approaches that learn the underlying distribution of normal data and detect deviations from it.

Click to watch →

Deep Dive

Basic proficiency in Python's data science stack (such as pandas and scikit-learn) to understand the implementation of practical machine learning pipelines.

Como criar Pipelines em Python

Como criar Pipelines em Python

13.1K

A Pipeline in Python's scikit-learn is a sequential combination of data transformation steps followed by a final estimator, which organizes and executes preprocessing operations (such as scaling, imputation, and encoding) before model training, ensuring consistent application of transformations across training and testing datasets while improving code organization, reproducibility, and preventing data leakage.

Click to watch →

Scikit-Learn Model Pipeline Tutorial

Scikit-Learn Model Pipeline Tutorial

34.6K

The scikit-learn Pipeline object enables chaining multiple data transformation and modeling steps into a single workflow, allowing different transformations to be applied to different features (such as using StandardScaler for some columns and MinMaxScaler for others) before fitting a machine learning model like Linear Regression, K-Nearest Neighbors, or Random Forest, thereby streamlining the machine learning pipeline process.

Click to watch →

Machine Learning Pipeline In Python | How to run pipeline in python machine learning

Machine Learning Pipeline In Python | How to run pipeline in python machine learning

19.4K

A machine learning pipeline in Python combines data preprocessing steps (such as MinMaxScaler for scaling and PCA for dimensionality reduction) with model training (like Logistic Regression, Decision Tree, or Random Forest) into a unified workflow using scikit-learn's Pipeline module, enabling more manageable, portable, and debuggable code while allowing easy comparison of multiple models on the same dataset.

Click to watch →

Building a Machine Learning Pipeline with Python and Scikit-Learn | Step-by-Step Tutorial

Building a Machine Learning Pipeline with Python and Scikit-Learn | Step-by-Step Tutorial

31.2K

Machine learning pipelines in scikit-learn organize preprocessing steps so that the output of one step becomes the input for the next, preventing data leakage and enabling workflow reuse; basic pipelines use make_pipeline for simple transformations like imputation and scaling, while advanced pipelines for mixed numeric and categorical data require ColumnTransformer to apply different preprocessing strategies to different feature types before combining them with a classifier.

Click to watch →

Learn Machine Learning Like a GENIUS and Not Waste Time

Learn Machine Learning Like a GENIUS and Not Waste Time

1.1M

Scikit-learn is the number one machine learning library in the world for basic machine learning algorithms. A basic scikit-learn tutorial can be completed in one or two days. Its simple and consistent syntax means once you know how to use it for one algorithm, you know how to use it for any algorithm, provided you understand what each algorithm is meant for. The library comes with great documentation and toy datasets for practice. It is recommended to start using scikit-learn while learning about algorithms in the statistical learning course, implementing algorithms in three ways: from scratch using basic Python, using scikit-learn, and applying both approaches to real datasets.

Click to watch →

Intro & Scope

Anomaly Basics

Banking Applications

Detection Challenges

Isolation Forest Core

KDD99 Performance

Parameter Robustness

AML Data Synthesis

Practical Strategy

Conclusion & QA

Intro & Scope

0:08

Playing Section

1
Speaker introduces talk on isolation forests for AML anomaly detection.
2
Outlines topics: anomaly types, banking applications, and algorithm details.
3
Notes focus on unsupervised methods due to lack of true labels.

Deep Dive

Alternative and advanced unsupervised anomaly detection algorithms, such as Local Outlier Factor (LOF), One-Class SVMs, and Deep Learning-based Autoencoders.

40 Algorithms in 40 Minutes

40 Algorithms in 40 Minutes

371

This final section covers advanced unsupervised learning and deep learning techniques. Clustering groups similar data points without labeled examples using distances or differences. K-means iteratively assigns points to nearest centroids and updates centroids as means. Apriori algorithm discovers association rules in transactional databases using support, confidence, and lift metrics. Mixture models represent populations as combinations of subpopulations using probabilistic distributions identified via Expectation-Maximization. DBSCAN clusters based on density using epsilon (neighborhood radius) and minPts (minimum points for dense regions). OPTICS extends DBSCAN with reachability plots showing clustering structure at multiple density levels. Hierarchical clustering builds dendrograms showing relationships at multiple scales. Anomaly detection identifies unusual data points differing from the majority. Local Outlier Factor (LOF) identifies outliers by comparing local densities. Autoencoders learn efficient data representations for dimensionality reduction. Deep Belief Networks use multiple hidden layers trained layer-by-layer. Hebbian learning, inspired by biological neurons, adjusts synaptic weights based on correlated activities for cognition development.

Click to watch →

Anomaly detection (Local Outlier Factor)

Anomaly detection (Local Outlier Factor)

209

Anomaly detection identifies data points deviating from normal patterns by learning normal data distributions. Applications include quality defect detection, system error monitoring, and financial fraud detection. Major algorithm categories include density-based methods (LOF), model-based methods (Isolation Forest, One-Class SVM), and reconstruction error-based methods (PCA, Autoencoder). LOF specifically analyzes local data density: normal points share similar local densities, while outliers exhibit lower densities relative to neighbors. The algorithm systematically calculates k-distance to define neighbors, computes reachability distance considering local density, derives local reachability density, and finally determines LOF scores to identify outliers.

Click to watch →

AI Bootcamp 2020- Anomaly Detection: Prof Thomas Dietterich

AI Bootcamp 2020- Anomaly Detection: Prof Thomas Dietterich

230

Beyond density estimation, alternative approaches include: one-class SVM finding decision boundaries containing (1-ε) of data; Local Outlier Factor normalizing nearest neighbor distances by local density; Mahalanobis distance accounting for feature correlations via inverse covariance; and random forest similarity scores capturing higher-order interactions. Projection-based methods transform data into lower dimensions for robustness: Isolation Forest builds random trees recursively measuring isolation depth (splits needed to isolate a point)—outliers require fewer splits. LODA creates sparse random projections (3 features each) and fits one-dimensional density estimators, averaging scores across projections. Both methods gain robustness through feature subset focusing, reducing vulnerability to irrelevant dimensions and nuisance novelty.

Click to watch →

Local Outlier Factor (LOF) for Anomaly Detection | Unsupervised Machine Learning

Local Outlier Factor (LOF) for Anomaly Detection | Unsupervised Machine Learning

5.1K

The Local Outlier Factor (LOF) is an unsupervised machine learning algorithm that detects anomalies by comparing the local density of each data point with its neighbors; it distinguishes between anomaly detection (where outliers are included in the training data) and novelty detection (where only normal data is used for training), with the key difference being that anomaly detection identifies outliers within the training data while novelty detection identifies new outliers in test data.

Click to watch →

Process Fault Detection using One Class SVM and Isolation Forest || Part-7

Process Fault Detection using One Class SVM and Isolation Forest || Part-7

1.2K

This section introduces two popular unsupervised anomaly detection algorithms: One-Class SVM and Isolation Forest. One-Class SVM creates a decision boundary around normal data using maximum margin classification with RBF kernel transformation. Isolation Forest detects anomalies by measuring how easily data points can be isolated using random splits, where anomalous points require fewer splits. The video also provides an overview of other outlier detection methods including robust covariance, local outlier factor, and autoencoders available in Python libraries.

Click to watch →

Deep Dive

Evaluation methodologies for highly imbalanced datasets, including Precision-Recall curves and Cost-Benefit analysis rather than standard accuracy metrics.

Scikit Learn Workshop with Andreas Mueller 3-30-2015

Scikit Learn Workshop with Andreas Mueller 3-30-2015

6.5K

Accuracy is misleading for imbalanced datasets because a classifier could achieve high accuracy by always predicting the majority class. Alternative metrics include: (1) Area Under the ROC Curve (AUC-ROC) - measures ranking quality by comparing true positive rate against false positive rate at various thresholds; (2) Precision-Recall curves - more informative than ROC for highly imbalanced problems; (3) F1 score - harmonic mean of precision and recall. AUC-ROC is particularly valuable because it evaluates the classifier's ability to rank positive instances higher than negative ones, regardless of the chosen decision threshold. Scikit-learn provides these metrics through the metrics module with string identifiers for easy specification in cross-validation and grid search.

Click to watch →

Lecture 11 | ROC Curve | Precision Recall Curve | Area Under Curve | AUC | Explanation

Lecture 11 | ROC Curve | Precision Recall Curve | Area Under Curve | AUC | Explanation

270

This section introduces Precision-Recall Curve as the preferred evaluation method for imbalanced datasets. Precision-Recall Curve plots Precision on the y-axis and Recall on the x-axis at multiple threshold values. Precision measures positive prediction quality (TP/(TP+FP)), which is more relevant when positive examples are rare. At threshold = 0, all points are predicted as positive (recall = 1, precision = 0). At threshold = 1, all points are predicted as negative (recall = 0, precision = 0). An ideal model should maintain high recall while increasing precision as threshold increases. AUPRC (Area Under the Precision-Recall Curve) is used to compare models for imbalanced datasets, with higher AUPRC indicating better performance at detecting positive classes.

Click to watch →

Webinar "Evaluating XGBoost for balanced and Imbalanced datasets"

Webinar "Evaluating XGBoost for balanced and Imbalanced datasets"

3.1K

For imbalanced datasets, accuracy is fundamentally misleading and should never be used. Recommended evaluation metrics include: Precision-Recall Curve (PRC) showing trade-offs between precision and recall; F1 score (harmonic mean); F0.5 (emphasizes precision); F2 (emphasizes recall); Matthew Correlation Coefficient (MCC); and cost-based evaluation. The baseline PRC equals the positive class proportion. Examples demonstrate why accuracy fails: a classifier predicting all negatives achieves high accuracy but zero recall; a classifier predicting all positives achieves high precision but low recall. The choice between precision and recall depends on application context—for medical diagnosis, avoiding false negatives is critical; for customer blocking, avoiding false positives preserves customer relationships.

Click to watch →

Precision-Recall

Precision-Recall

6.4K

The Precision-Recall curve is a performance evaluation metric for classification models that addresses the limitations of the ROC curve in imbalanced datasets; unlike ROC which uses false positive rate (biased by negative class size), Precision-Recall uses precision (true positives divided by predicted positives) which remains unbiased by class distribution, making it more suitable for scenarios with rare positive classes such as medical diagnosis where minimizing false positives is critical.

Click to watch →

Mastering MLOps: From Model to Production

Mastering MLOps: From Model to Production

1.3K

For fraud detection with heavily imbalanced datasets (fraud rates often below 0.1%), accuracy is unreliable because trivial models predicting 'not fraud' achieve high accuracy without usefulness. Appropriate metrics include: Precision (proportion of predicted fraud that is actually fraudulent), Recall (proportion of actual fraud correctly identified), F1 Score (harmonic mean of precision and recall), and AU-PRC (Area Under Precision-Recall Curve). ROC-AUC is less effective due to the large denominator of true negatives overwhelming variations in numerator values.

Click to watch →

Deep Dive

Model explainability techniques (like SHAP or LIME) to interpret why specific high-risk transactions were isolated, which is a key regulatory requirement in AML.

Agentic AI for Anti-Money Laundering

Agentic AI for Anti-Money Laundering

6.1K

Rules-based AML systems generate numerous false positives that consume significant institutional resources. Common triggers include many-to-one transaction patterns (which could indicate drug collection or simply landlord rent collection), large round-dollar payments (reflecting legitimate business transactions like foreign exchange conversions), and other patterns that trigger alerts but represent normal activity when contextualized. The greatest value of AI in AML lies in reducing false positive burden, allowing human investigators to focus on high-value work. Currently, most AML professionals spend 7-8 hours daily resolving false positives, leaving little time for actual suspicious activity investigation. Financial institutions face significant challenges with AI explainability because regulators require clear justification for decisions. Unlike traditional rule-based systems, AI models operate as black boxes, making it difficult to demonstrate reasonable steps taken to detect and prevent money laundering. Recent advances enable LLMs to generate clear explanations and summaries, helping institutions satisfy regulatory scrutiny.

Click to watch →

SCOR Foundation - AI and Internal models : the case of banking"

SCOR Foundation - AI and Internal models : the case of banking"

178

ML interpretability faces fundamental challenges in high-risk banking applications: (1) Post-hoc methods (SHAP, LIME) provide approximations that may not reflect true model behavior and can produce inconsistent explanations; (2) Imbalanced data affects both model learning and interpretation reliability, particularly in low-default portfolios; (3) Different stakeholders require different explanations, and no single method provides perfect transparency. These limitations create barriers to ML adoption despite potential accuracy improvements, as supervisors require clear understanding of how risk parameters reflect actual risk levels.

Click to watch →

Explaining the unexplainable - can we depend on what we don’t understand? | GHOST Day: AMLC 2020

Explaining the unexplainable - can we depend on what we don’t understand? | GHOST Day: AMLC 2020

109

Explainable AI (XAI) methods like LIME and SHAP enable us to interpret complex, high-accuracy black-box models (such as neural networks) by providing model-agnostic explanations that work across different types of models, allowing us to understand feature importance, detect biases, improve model performance, and comply with regulations like GDPR that require explanations for automated decisions.

Click to watch →

Secure and compliant machine learning workflows with Amazon SageMaker

Secure and compliant machine learning workflows with Amazon SageMaker

5K

This section covers model explainability and interpretability for regulatory compliance. For tree-based models like XGBoost, use built-in APIs to extract feature importances or libraries like SHAP. Pull the model object from S3 into the notebook environment, unpack it, and use scikit-learn libraries to extract and plot feature importances. SHAP values can be calculated by computing two models (with and without each feature) and comparing outputs to assign importance values. This enables regulatory audits by visualizing and saving insightful metrics like feature importance and SHAP values directly from the notebook.

Click to watch →

[Webinar] Translating ML to Business Value in Financial Services

[Webinar] Translating ML to Business Value in Financial Services

302

There are three main approaches to model explainability: (1) Inherently interpretable models using classical statistical methods or constrained ML algorithms; (2) Post hoc explainability techniques like LIME and SHAP values applied after model training; (3) Counterfactual explanations telling users what they would need to change to get a different outcome. The regulatory landscape presents challenges: regulators want to encourage ML innovation but cannot provide specific guidance on what 'explainable' means. The CFPB guidance states that complex ML models are not an excuse to avoid providing turn-down reasons, but doesn't prescribe how to achieve explainability. Organizations must balance innovation with compliance while maintaining stakeholder buy-in.

Click to watch →

Deep Dive

The practical challenges of deploying anomaly detection systems in production, such as handling real-time streaming data, concept drift, and managing false positive rates.

Generic and Scalable Framework for Automated Time-series Anomaly Detection

Generic and Scalable Framework for Automated Time-series Anomaly Detection

3.2K

This section addresses the practical realities of deploying anomaly detection systems in production environments. Evaluation methodology uses F-score against open-source datasets, revealing no single model performs optimally across all cases—justifying the framework approach. The team created and open-sourced their own labeled anomaly dataset since none existed publicly. Key deployment challenges include reducing false positives for user adoption, handling non-uniform hardware in server farms, and distinguishing anomalous usage from system failures. Univariate detection generates false positives when analyzing heterogeneous systems; multivariate approaches using convex hull ranking reduce these by comparing metrics relative to each other. The system automatically trains thresholds to minimize false positives/negatives and incorporates user feedback through generalized boosted regression models. Critical distinctions involve recognizing when increased traffic causing higher response times represents normal usage versus when disproportionate increases indicate true anomalies. This creates a self-improving system where user preferences refine detection over time, enabling practical deployment at scale.

Click to watch →

#157 - Low noise threat detection with Joshua Neil, Founder at Alpha Level

#157 - Low noise threat detection with Joshua Neil, Founder at Alpha Level

691

Even with path-based anomaly detection, the false positive rate remained too high for practical use. With only about 10% true positive rate, nine out of ten alerts were false positives. Since net flow data is very voluminous, this created an unmanageable number of alerts for tier one security operations. This revealed that while anomaly detection is good at finding new threats, the false positive problem must be addressed separately to make the technology usable in production environments.

Click to watch →

When Seconds Cost Millions: Streaming Data for Instant Anomaly Detection

When Seconds Cost Millions: Streaming Data for Instant Anomaly Detection

457

Building anomaly detection applications involves defining patterns programmatically (e.g., minimum 10, maximum 30 anomalous data points) and defining response logic that summarizes events into results including timestamps, IPs, and metrics. Applications run locally for debugging using secure tunnels to AWS clusters, then deploy to Managed Flink. Deployment involves uploading compiled JAR files to Amazon S3, configuring service roles for IAM integration, setting up auto-scaling, checkpointing for durability, and configuring logging to CloudWatch. The managed service handles versioning, scaling, and operational overhead. The Flink dashboard shows pipeline status and processing metrics. Operating streaming systems at scale presents challenges: customers need time to build operational expertise, failures must be managed carefully, and understanding Flink configurations for durability and performance is complex. The core primitives for any streaming architecture are windowing, stateful processing, real-time event consumption, and rolling state management. These enable sub-second detection across millions of events for fraud detection, infrastructure monitoring, and security applications.

Click to watch →

Real-time anomaly detection on streaming data using Random Cut Forest on Apache Flink

Real-time anomaly detection on streaming data using Random Cut Forest on Apache Flink

376

Anomaly detection identifies data points or patterns deviating significantly from expected baselines. Two primary types exist: point anomalies (single data points far from the mean, detectable individually) and collective anomalies (pattern changes requiring temporal analysis of neighboring points). Critical to accurate detection is contextual understanding—mathematical models alone cannot distinguish true anomalies from expected variations. Real-world implementation faces three fundamental challenges: lack of labeled training data (requiring unsupervised approaches), concept drift causing baselines to shift over time, and partial data sets meaning models must operate with incomplete information. The implementation workflow involves ingesting streaming data (e.g., via Kafka), processing through anomaly detection operators, and storing results for downstream analysis. Success requires balancing mathematical rigor with practical operational considerations.

Click to watch →

Piotr Guzik - REAL-TIME ANOMALY DETECTION MADE EASY

Piotr Guzik - REAL-TIME ANOMALY DETECTION MADE EASY

915

Production deployment requires addressing asymmetric anomaly responses where traffic gains may warrant investigation while losses demand immediate attention. Systems must distinguish between transient issues and long-lasting changes that have become new normal behavior. Organizations should define maximum anomaly duration thresholds beyond which detected anomalies are no longer actionable. For example, if a data loss anomaly persists for six hours without resolution, it transitions from being an alertable issue to a documented operational reality requiring different handling.

Click to watch →

Intro & Scope

Anomaly Basics

Banking Applications

Detection Challenges

Isolation Forest Core

KDD99 Performance

Parameter Robustness

AML Data Synthesis

Practical Strategy

Conclusion & QA

Intro & Scope

0:08

Playing Section

1
Speaker introduces talk on isolation forests for AML anomaly detection.
2
Outlines topics: anomaly types, banking applications, and algorithm details.
3
Notes focus on unsupervised methods due to lack of true labels.

94.9K views1.3K32:47@PyDataTVOriginal Release: 2018-05-28

Isolation Forest is an ensemble-based unsupervised anomaly detection algorithm that excels in financial crime compliance contexts like money laundering detection, where supervised learning is impractical due to lack of labeled data; unlike traditional distance/density-based methods, it isolates anomalies by building extremely randomized decision trees and calculating anomaly scores based on the average number of edges traversed to isolate each data point, making it robust to parameter settings and effective for detecting subtle anomalies hidden in normal transaction patterns.

[00:00:08]hello everyone thank you for well welcome my name is Elena Neves scientist at create Swiss I work in a financial crime compliance team and this talk is about using isolation forest for anomaly detection with application to you AM L AM L stands for antimony wondering can I have a quick show of hands of who in the audience works with in financial crime compliance or otherwise in the in this area okay that's good so in this talk I will briefly introduce what is anomaly detection and what kind of methods are used we'll look at the application of anomaly detection to banking so Walker will cover the three most often used areas then we'll talk about specifically for AML if supervised purses and supervised methods are applicable and then we'll look at the isolation forests so we'll step away from the topic of AML and look at the actual algorithm I'll describe police resolution first I will describe how it's implemented the Sackett learn and we'll compare performance on the public data set key to carp 99 most of you probably have used that before and will compare the performance to a local I'd laugh actor then we'll move to isolation force performance on a synthetic data set that has been put together specifically with a few antimony ml red flags embedded in it and then we'll conclude here's a quick disclaimer for you to read okay so what is an anomaly anomaly is a data point of interest in in this case it's something that stands out it's it's an unusual an unusual data point when a data generating process behaves and usually it results in an anomaly anomaly as an anomaly detection as a topic is a well researched subject there is a lot of publications out there in basta testicle and data science the real challenge for us who practice data science is to essentially define construct the right data model for the subject at hand to separate outliers from noise and from from the actual normal data what kind of methods are used well the methods are broadly speaking fall into three categories the density based methods the distance based methods or parametric you probably used most of these before the density based and the distance based they fall into the spacial proximity type of algorithms and the examples here are DBC and vocal a factor K nearest neighbor k-means or something like a distance to a regression hip of languages in regression the parametric passes they would usually assume some sort of form to your data so assuming that when you're working with so what you have observed so far is normal and models like garden mixture model or single classes Williams I use there of course there are other methods which are not kind of machine learning methods like z-score which will assume normality and many more in terms of application to banking so many of you probably have heard of anomaly detection is used in credit card for detection that's a very very well-known area where it's used other other areas such as market abuse and private banking so much taboos by clients Oh market abused by both clients and traders in investment banking I'm highlighting the ml as the money laundering as a subject of detection for all three retail private and Bank and what is important to understand is that the the way manual wandering is done using these three types of banks is actually different and therefore different methods of detection and different types of red flags exist in in in these banking types so here if you learn one thing from this talk this should be the one so another point to be aware of is that also mate is money learning detection is very it's actually an inherently different problem from optimizing detection tradecraft cray-pas forward or automatic detection of market abuse and why is it so well in credit card fraud we often well we have a luxury of actually having data when we have true positive cases or labels and why because often customers worksheet tell us if certain transaction wasn't actually done by them it market abuse it's slightly hard but still you have you have a signal in the data so for example the P&L positive P&L or huge changes in piano or huge changes in the price move can be used as a self-revealing class in case of automatic money laundering detection it's much harder because most of the times especially from the from the banking perspective from the banks we don't really have any data that can be used as a true positive set of cases or labels what banks do they if they suspect money laundering they will file a star to the National Crime Agency and often it may actually never hear back from them or if they do here it takes years so it's not really practical so for detecting money laundering for for banking we're pretty much left with just unsupervised methods so why is it difficult well the royalty papers they're actually quite well described why is it difficult but of course there is severe class imbalance this is an outlier detection problem what we also deal with Steve Mira class overlap what that means is that money-laundering is is when it comes from a single stream of financial transactions actually be mixed with with normal normal activity so what criminals do they will use front companies sometimes banks and it makes it much harder to detect another thing that we have to deal with is called concept drift so how money is laundered even by the same criminal organization changes over time and of course there is our certainty around the day tomorrow so if you choose to use your suspicious activity report as an indication for what what whether certain transaction of such transactions were actually money laundering then you have a lot of uncertainty as Angelus so you have a picture of a candle in the dark most of the time there is actually no candle either so let's step aside from the problem of money laundering and let's look at the isolation forest isolation forest is an ensemble regressor and it is different from other types of distance or density based method is that it doesn't profile normal cases and it doesn't calculate distances foreign-based distances for the food item instead what it does it essentially tries to separate away data points or explain away anomalies from normal normal data points underneath it is building extremely randomized decision tree and it uses that as a base estimator isolation forest why is so good for anomaly detection well because of campus work post as a supervisor or an unsupervised classifier it calculates an anomalous score which is 2 to the power of minus a ratio of expected number of edges in the trees for in the trees in the forest for particular data point normalized by a constant in the supervised setting you are going to use a threshold which is a contamination ratio to to create a binary classification problem in the if you can actually use use it without the threshold and turn it into a soft classifier so as the number of edges the average number of edges in the trees goes to zero the higher is the anomaly score so the reason is that actually anomalous data points they are easier to explain away from normal data points the the model isolation force was introduced in 2008 and became available in scikit-learn in 2016 so in the previous version to the current one it extends the booster activity regressor and it's possible to use it as both with and without replacement by default is without and it's as I said already it's based estimator is the extremal random strain regret sir so both the attribute to split and the displease value will be chosen randomly this allows you to overcome building very greedy trees in practice in the implementation circuit lien it will return a value of 0.5 minus the anomalous score so smaller scores or negative scores will be indicative to two anomalies so let's take a look at the performance of this this classifier on a public dataset kgt up 99 we're going to quickly switch to hopefully you can see so I'm going to use schizocarp 99 and this will be downloaded from the secular datasets module and there are a few options for downloading this modulus let me just briefly scroll through the helper methods and about the datasets so we'll use SI + SF dataset sa has only four four attributes and herself has 41 the kg Cup 1994 so don't know contains loads of offline intrusion detection system it's been around for quite a while the SF anomaly rate in this case is 0.5% so I point 5 percent of data points will contain actual attempts of hacking attempts and I say I'm only loading 10% of the data and here the anomaly rate is higher it's three point four percent so we're going to perform some pre-processing so what we're going to do is we're going to save all of these all of the attacks and turn them into one class and everything which is not an attack will be probably a negative class only states a positive class and for the sake of local at live factor I've also normalized the data and fields that are not numeric fields have been turned in numeric using the label and folder in terms of training isolation forests and the local life factor we're going to split the test set and the training set at 33% and if you look at the setting the features I'm pretty much using the default recommended features number of estimators I'll talk about what this parameter does number of estimates is a hundred we will use in this case because we have a label in in the data set will use contamination rate of 15% so much higher than the natural rate of anomalies in this case and max sample set 25 okay so it takes about 20 seconds to Train only as this is the SI data set so this is a smaller one and when a normalized States a local I fact the training is faster but let's you take a look at the results so I'm going to show you the output of the of the year classification report as well as the confusion matrix in terms of anomaly detection what we are always interested in is we are really interested in the recall which is which is this upper left-hand corner and you can see that only trainings that this relation Forest performs actually quite well and it gets about 12 percent false positive right I also calculate I you see also in this case it may not always be relevant for you and it's a pretty high I will see the local knife factor on these dates that doesn't do very well it only achieves a recall rate of 8 and a half percent and much higher false positive rate so one more question so why is no qualifier fact that doesn't do very well perhaps it's the number of attributes in this data set is quite a lot is 41 and it may be it's just getting the dimensionality Kherson here so let's take a look at the SF data set some data set has only four attributes and it's a slightly bigger one in this case again we are going to split test and reset three percent and we're going to test a solution force was exactly same parameters take slightly longer to train and the results are very similar to the previous set so it's eves very high recall which is exactly what we're interested in and not a very high false positive right I can look at the fact that didn't do very well perhaps it's the dataset itself I've also tested it on non normalize dataset the log of that factor and I didn't see much difference so in this case my well might say that you might play with the number K which is which is irrelevant k nearest neighbor setting for local of that factor in order to try to get a perform better but this talk is not about LOF so let's move on it never performance on the test set what is surprising is that isolation forest does work pretty well and tested as well so we have a 99% recall very similar to the training set and a hundred percent for the SF okay so now let's talk about the estimators what I find is that isolation forest is actually very robust to parameter setting and estimators will control how many trees get built in the isolation forest and we're going to see how sensitive it is to this parameter for the sake of this particular test I'm turning my my scorer into a recall score and I turned the minus 1 so the anomaly into a positive label so we'll test the performance on any where estimated so number of trees in forests between 20 to 230 and what we see is that both recall and I see a quite insensitive to this and this actually matches to what the original or have published in their paper about the algorithm being very robust after about 20 it achieves a very good recall mark samples will control how much of the data gets sampled to build the trees in the other nation forest and the idea here is that it's the sub sampling that helps to explore these hidden layers or subsets of the data so we'll look at the performance of isolation forest always max sample between 10% to going up to 100 so what we see is that it does serve starts to suffer if you increase sampling size and it is actually also something that resonates back to the original paper now the contamination rate so if you choose to use a solution forest as a supervised to classify it was used the contamination rate as a threshold to define on a set of all the anomaly scores for all the data points were to cut off in order to make it anomaly versus not anomaly so this is the parameter that the algorithm is actually quite sensitive to and you can see that we are going to test it for anywhere between 1% so this is still higher than the natural contamination rate to 25 and after about 10 percent of contamination it achieves a recall of 100 so we have looked at in terms of parameterization so I said it's a it's it's pretty robust to parameters and in complexity it compares quite well complexity to K K nearest neighbor and other distance based or density based algorithm so it's pretty you could say that it's linear in its complexity so let's go back to the topic of money laundering I have said I started this talk by saying what is an outlier what is an anomaly but the truth is that most of the automated money laundering detection is actually not a simple anomaly detection problem and it's not really an outlier detection problem and why is that well it's because many hundreds of transactions that are associated with money laundering they differ very little from normal transactions so as Liars are they are hidden in the unusual behavior of low dimensional subspaces and if you do learn a second thing from this talk this should be the one so I have put together a data set which is well which is a synthetic data set it's made up of fake transactions but it has some money laundering red flags embedded into it the first Jupiter notebook is available for you from my github page and I can share with you the second Jupiter notebook if you want to you can contact me through my blog the link is at the bottom so the idea here is that we we have to take our clients transactions and trades and we have to essentially optimize it for a set of red flags the red flags they are they there are publications about them your local financial financial crime subject-matter experts will know more about them but if you if you're interested to read than consult financial action task force website or DFS or join my new learning group of walls back row so I'm going to switch to my synthetic data set and I'll talk about it a little bit more the synthetic data set is very small in this case it has only 200 data points and I have created four strategies in there and first strategy the anomalies essentially at five percent rate so what does it look like it looks about like this what we've put in the data point in in the data set we have a client ID we have a made-up security name it can be anything a stock or bond or anything each client in this case has ten transactions one transaction per day we have a notional amount which is purely random source from uniform distribution we have a flag for whether to buy or sell we have a flag for whether this transaction is followed by an asset transfer so if it's a pile and it may be followed by an asset transfer out if it's a sell it may be followed by an asset transfer in sorry preceded by a asset transfer in and then there is some sort of currency pay in this case were not looking at the currency pair of the foreign exchange type of strategy but this is a fake currency pair in most cases and we also have a flag for settlement to offshore legal entity so most of you who work with transactional data will already immediately recognize that this is one step or many a few steps after the raw data set so in this case what I'm what I'm kind of getting to is to the idea of summarizing no data for a specific type of off anomaly over over at flag so our niska's isolation force max sample I'm setting to 50% because this is a very small data set and what we're going to do is we're just going to feed the raw synthetic data into isolation forests in this case and after doing that we will compare what is the mean anomaly score for those records that were created to mimic a particular money learning strategy versus those that haven't and what we see here is not isolation forest does managed to assign a more negative score on average two data points that were designed to mimic money laundering visually this will look like this so these are the points this is the first set of trades sorry this is the first set of race so this is index 0 and mirror out mirror Ian and then offshore also isolation forest manages to get quite a few false positive in this case so let's go one step further we are now going to actually do what we said we're going to fold our data set and to optimize it for looking for a particular round flag in this case was trading I'm not going to talk about what it is if you want to know more about the strategies talk to me after this call after this presentation so we will summarize activity as a ratio of bias versus cells in the same security and this is what gets fed into the pass elation forest again was very much the same the same parameter settings in this case were not using contamination rate this is not a supervised classification problem and in my de said we only had one one example which was a true positive for sure and isolation forest dust find this data point let's look at another strategy as a transfer strategy so if you go on a DFS website and search for Mira we will find documents that are specific to this this is a very well known case and what we are looking for we are looking for activity where which is preceded or followed by as a transpose this is what gets summarized I folded into the summary and gets fed into the isolation for us there were two data points not two data points two trading strategies at all 20/20 records in a data set and they are index 1 and the index so again isolation forest does manage to find it so far I hope you understood that the idea for using anomaly detection algorithm is to be able to explore these sub samples of a data set and another extension to application to anomaly detection would be to use what's called a risk-based approach which is often what the financial crime compliance teams tend to do but the idea here is that to actually focus on a particular group of high-risk clients focus on particular client network for example all legal entities that share the same beneficial owner or all legal entities that actually share the same big code and anything else and for the other so and then the this this extra parameter this extra feature will be then introduced into the isolation forests so in conclusion the anomalous activity in AML is Bank type specific different strategies are used for different types of Bank isolation Forest is a promising robust classifier that can be used for in within the AML space the successful automatic detection starts with asking the right question and then optimizing the data set for the data model that that you want to construct thank you very much [Applause] [Music] any questions thank you very much for the presentation I'd be interested so an isolation forest doesn't scale terribly well typically doesn't do out of core so I'd be interested if in the applications you're interested in that's a challenge both in terms of memory or in terms of CPU usage so as I said the quite often financial crab plants team they will focus on a particular risk or group of clients in order to do a risk based approach that in itself minimizes the type of Records you have to look through and another thing is that unlike in prey crowd for detection in my eulogy detection we have a luxury of doing it offline it doesn't have to be immediate or as fast as possible it can take you can take a few days or weeks in order to do it offline so typically the problem is the memory usage so can I ask whether you're using like hardware where the large large amount of memory or not the reason I'm asking is that people often tell us it's too expensive to buy hardware with one terabyte of RAM and then I get a fight worthy where the person's like it's not that expensive and I just can't do isolation forest out of court so do you run specific hardware or just small computers yeah and firstly I cannot us answer the the specifications of the machine that we use at the bank I didn't put it together I know it's a supercomputer yeah thanks to that you so initially you showed really good recall it was sort of in the demonstration really good recall but quite poor LOF I'm not really familiar with that we'll just have a quick look now it's like presumably just something for anomaly detection can you just give some more intuition like why do you think that is or because it sounds like you're spreading quite a wide net but then you're missing possible anomalies so I don't know that's what it seemed like maybe yeah I'm not sure exactly why LOF didn't do very well on this task I think it's just this particular dataset it doesn't perform very well I have seen very similar performance of LOF on this data set reported in other papers so I just gave up and try to optimize it I think it's the it's the fall of decay the number of neighbors is very it's very sensitive to that parameter and I think that's probably where if you wanted to optimize if you would you would play with that parameter Thanks would you be able to say anything more about the metric which isolation forest is using you mentioned about the number of edges about the metrics about the about the expectation number of edges number of edges it's sorry if the question it's not very clear the metric question okay so what happens is that isolation forest has two stages the training stage and the testing stage in the training stage it will build the trees in the in the in the forest and in the testing stage it will pass each data point through each tree and it will code it will record at what stage hits the external load and then the number of edges it has to travel through for each tree gets averaged over and the shorter is that traveling through for a data point the more likely it is to be the anomaly I hope their answers thank you very much for the talk quick question so because the datasets have labels that you that you can use to calculate recall and metrics how do you apply that in practice when you don't have the labels and how do you tune your model since you can't you know calculate recalling metrics yeah you're right in practice we don't have labels and so what we have in the financial Chrome team we have analysts who the cases that are seemed of interest they get followed forwarded to and they essentially have it's it's it's a humans looking at transactions and trades and trying to understand whether they are of interest or not that's how it's used in practice oh I think you mentioned that false-positive rate is about 12% right but in practice what can you do if you have such a high positive false positive rate I mean you probably get referred such a large amount of cases and what can you do how can you look through all of that for young yeah it's actually a false positive rate of 12% for AML is actually very low most of the time most of the time the false positive rate is close to 100 so it's very promising thank you [Applause]

Up Next

Isolation Forest Anomaly Detection in Python & Scikit-Learn

@alisterluiz

9.2K views•2024-06-22

Building Real-Time ML Pipelines with Feature Stores and MLOps Frameworks

@ODSCAI

5.1K views•2022-02-20

Neural Networks for Recommender Systems (PyData 2017)

@PyDataTV

21.1K views•2017-04-25

Neural Networks Explained: Math, Layers, and Learning Fundamentals

@3blue1brown

21.9M views•2017-10-05

Related Study Plans & Knowledge Roadmaps

Structured learning paths in Artificial Intelligence

View All Artificial Intelligence Topics

artificial-intelligence

Building Recommenders: Two-Tower & Vector DBs

01Foundations of Python & Machine Learning Basics

02Introduction to Recommender Systems

03Deep Learning & Representation Learning

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

Latent Diffusion: PyTorch, UNet & CLIP

01Deep Learning & PyTorch Foundations

02Convolutional Networks & Autoencoders

03The Math and Mechanics of Diffusion Models

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

NeRFs: 3D Reconstruction & Instant-NGP

01Math & Python Foundations for 3D Vision

02Photogrammetry & Camera Pose Estimation

03Neural Radiance Fields (NeRF) Core Concepts

+ 2 more learning modules

Explore Roadmap

artificial-intelligence

TFT Forecasting: Energy Demand & Deep Learning

01Programming & Data Science Foundations

02Introduction to Machine Learning & Time Series

03Deep Learning & Sequential Models

+ 3 more learning modules

Explore Roadmap