Anomaly detection identifies data points, events, or patterns that deviate significantly from expected behavior within datasets, and in the SAP ecosystem, this is achieved using SAP HANA ML tools including the Predictive Analysis Library (PAL) and HANA ML, which offer algorithm interchangeability (DBSCAN, Isolation Forest, One-Class SVM, K-Means), out-of-the-box features for time series and clustering, and native in-database processing with enterprise-grade security; successful implementation requires proper data preparation (null handling, feature importance analysis, dimensionality reduction), appropriate algorithm selection based on use case characteristics, and systematic evaluation using metrics like precision, recall, F1 score, and ROC-AUC for labeled data, or domain expert validation and score distribution analysis for unsupervised approaches.
Anomaly Detection with SAP HANA ML: Best Practices for BTP
Added:Welcome to this session on Anomaly Detection.
As part of SAP BTP AI best practices, we'll explore how to efficiently identify unusual patterns and data using SAP HANA machine learning tools.
By the end of this session, you'll have a clear understanding of key principles, best practices, and implementation approaches to detect unexpected behaviors within your data.
Let's get started.
Our discussion will follow these four key steps.
First, we'll provide an overview of anomaly detection and its benefits in the SAP ecosystem.
Second, we'll cover the essential prerequisites you'll need to have in place, including supported environments and required components.
Third, we'll explore key choices and guidelines that impact your implementation, including algorithm selection and parameter optimization for different use cases.
Finally, we'll dive into implementation approaches across programming models, providing concrete examples for Python, SQL, and R.
Anomaly detection is the process of identifying data points, events, or patterns that deviate significantly from expected behavior within data sets.
In the SAP ecosystem, this involves leveraging SAP HANA ML tools, specifically the Predictive Analysis Library and HANA ML.
The expected outcomes include successfully identifying unusual behavior, or outliers, and various data types, enabling proactive responses to potential risks through early detection, optimizing business processes by identifying inefficiencies and unexpected variations.
Why use SAP HANA Machine Learning for anomaly detection?
There are three main benefits.
First, algorithm interchangeability.
You can easily switch between different algorithms like database scan, isolation, forest one class, support vector machines, and k -means to best suit your specific detection needs.
Second, out-of-the-box features that supercharge your development with built-in capabilities for time series, anomaly detection, clustering, and distance -based outlier scoring.
Third, security and SAP ecosystem integration.
It's fully integrated with SAP technologies, leveraging enterprise-grade security, and native in-database processing.
These advantages make anomaly detection more efficient and accessible for SAP users.
Before implementing anomaly detection solutions, let's cover the necessary prerequisites. You'll need SAP HANA Platform 2.0, SPS 04, or higher, or preferably SAP HANA Cloud for easier management.
SAP Datasphere or Express Edition can also be used for specific scenarios.
Application Function Library, or AFL, containing Predictive Analysis Library and Automated Predictive Library components. Script server enabled for executing machine learning algorithms.
Now, to provide more context, SAP HANA is an in -memory database enabling real-time analysis and applications.
The HANA ML libraries, Predictive Analysis Library and Automated Predictive Library provide native in-database functions for predictive analysis and machine learning.
The script server is an auxiliary HANA process essential for executing these machine learning functions.
Setting up these prerequisites establish a strong foundation for implementing effective anomaly detection solutions at scale.
Let's examine the high -level reference architecture for anomaly detection using SAP HANA ML.
The diagram shows the complete machine learning execution environment within the SAP HANA ecosystem.
Starting from left, we see end users and application clients connecting via HTTPS to access the system.
At the core is the SAP HANA database, which contains three key components.
Script server hosting, predictive analysis library, and automated predictive library libraries.
These provide the core anomaly detection algorithms, index server with our operator, and client capabilities for R-based analytics, external machine learning library for integration with TensorFlow and other frameworks.
The architecture includes external machine learning servers with our serverless runtime and TensorFlow capabilities connected through CloudHook for seamless integration for sharing use cases.
I'm sorry, streaming use cases. The streaming server supports continuous anomaly detection through streaming analytics.
This architecture ensures efficient, scalable, and secure anomaly detection while leveraging the performance advantages of in-database processing.
Let's discuss key choices that impact the performance of your anomaly detection applications.
Selecting the right algorithm is critical and effects detection accuracy, computational efficiency, and the types of anomalies you can find.
Different algorithms excel in different scenarios for general anomalies.
DBSync works best for a regular, non -convex shapes and explicit noise labeling.
Perfect for vessel GPS tracking to detect ******* phishing outside of designated zones.
Isolated forest excels with high -dimensional data and isolated anomalies.
Idea for credit card fraud detection with hundreds of transaction variables.
One class support vector machine is optional for complex decision boundaries and moderate-sized data sets.
It's effective for network intrusion detection.
K-Means provides clear explanations when working with well-formed clusters.
It's useful for call center metrics where performance data forms natural tiers for time series.
Time Series Anomaly Detection effectively handles seasonal data with pronounced trends.
It's excellent for retail demand with weekly cycles.
Using the right algorithm for your specific use case dynamically improves detection accuracy and system performance.
Algorithm performance is significantly affected by parameter settings.
When default values often produce good results, proper tuning ensures optimal anomaly detection.
Let's break down some of these key parameters for DBS.
MinPoints controls the minimum points required to form a cluster.
Higher values create more robust clusters, but may miss outliers.
EPPVS defines the maximum distance between points in the same cluster.
It's critical for determining what's considered close for isolation.
Forests and estimators determines the number of trees.
More trees increase accuracy but with diminishing returns.
Contamination effects.
Sensitivity to outliers represents the expected percentage of anomalies for one class.
Support Vector Machine Kernel Selection, RBF, or Polynomial creates effective non-linear boundaries, new controls, the upper bounds of the fraction of training errors, and supports vectors for time series, window size effects, smoothing, and sensitivity to recent changes.
Seasonality detection is critical for data with cyclical patterns like weekly or monthly trends.
Understanding and fine -tuning these parameters significantly enhances anomaly detection effectiveness.
Effective data preparation is crucial for quality anomaly detection in SAP HANA machine learning.
The guidelines focus on three strategic areas.
First, implement a clear null policy.
Define a strategy for handling all your missing values.
As many predictive analysis library procedures afford when encountering nulls.
SAV HANA Machine Learning offers tools like the Computer Within Predictive Analysis Library, providing methods like mean or median imputation.
Second, focus with feature importance analysis. When data has many features, some will be more useful than others for anomaly detection.
Use predictive analysis library techniques like feature selection or premutation importance to identify the most relevant features for anomaly detection.
This reduces noise, combats the curse of dimensionality, and improves accuracy. 3.
Simplify with dimensionality reduction techniques like PCA via predictive analysis library.
PCA function reduces features while retaining significant variance.
This improves efficiency and can make anomalies more apparent.
Systematically applying these techniques, no policy feature importance, and dimensionality reduction enhances anomaly detection reliability in SAP HANA.
Evaluating anomaly detection models is crucial, especially with unsupervised models, and SAP HANA machine learning offers tools for this assessment.
Our approach depends upon data and label availability.
Labeled anomalies use standard classification metrics like confusion matrix, true positives, negatives, false positives, negatives, precision, and recall. An F1 score can help balance false alarms and anomaly detection, which is vital for rare anomalies.
Finally, ROCAUC can measure the model's ability to distinguish between normal and anomalous instances.
SAP HANA and Machine Learning Predictive Analysis Library provides functions like confusion underscore matrix and AUC to compute these big metrics.
Unsupervised anomaly detection evaluation is more qualitative.
Domain expert validation is crucial.
Present flags anomalies with high scores to experts.
Perform score distribution analysis looking at the spread of anomaly spores to identify thresholds.
Understand interoperability, why a link was flagged for cluster methods.
Internal cluster validation indices like the silhouette score can offer insights into outlier separation.
Time series anomaly detection.
Visual inspection by domain experts is crucial for contextual evaluation if model fitting is involved. For example, in predictive analysis library outlier detection, TS review residuals to ensure model fit.
If labeled metrics are available, apply precision, recall, and AUC metrics using HANA and mouse tools.
As in the labeled anomaly analysis, the right evaluation methods depend upon the use case.
A combination of techniques provides the most comprehensive understanding of model performance in SAP HANA.
Let's discuss implementation guidelines for selecting your programming model.
For data science workflows, we recommend using Python with the HANA Machine Learning Library, providing a streamlined, intuitive experience aligned with standard data scientists' practices, including convenient data manipulation and integration with machine learning workflows.
For alternative approaches consider SQL script to directly call predictive analysis library procedures when tight integration with SAP HANA artifacts is needed.
Now let's explore the specific implementation options for each programming model.
For Python developers the HANA machine learning library is recommended for interacting with SAP HANA Predictive Analysis Library and the Automated Predictive Library Algorithms.
Reference code includes SAP BTP AI best practice sample code and SAP Community blog posts like learning from labeled anomalies for efficient anomaly detection using Python Machine Learning client or SAP HANA.
For SQL developers, direct invocation of PAL procedures through SQL script is available.
While embedding logic within SAP HANA artifacts is a priority.
Reference documentation includes comprehensive guides for disks, an isolated forest, one-class SVM, and other algorithms.
Each approach provides full access to anomaly detection capabilities while allowing you to work in your preferred environment.
The Python example demonstrates implementing the isolated forest algorithm using HANA ML.
We import isolated forests from the HANA machine learning library and create a model with default parameters.
Predictive analysis library automatically selects optimal values for trees, samples, and threads.
This simplifies configuration while maintaining performance.
The training process requires just a few lines of code.
Call the fit method with data key column and feature columns.
Use predict to identify anomalies in the data set.
The result is a data frame with anomaly scores for each record, making it easy to identify unusual patterns in your data.
This implementation showcases how quickly you can deploy powerful anomaly detection within minimal code through HANA ML. This best practice guide reflects the collective expertise of data science and analytics experts from across SAP, bringing together insights to support your anomaly detection journey. Thank you for watching this session on Anomaly Detection using SAP HANA Machine Learning.
We hope the information provided helps you successfully implement effective detection capabilities in your business applications to identify unusual patterns and prevent costly failures.
Up Next
Coded Bias: Racial Bias in Facial Recognition & AI
@DemocracyNow
47.4K views•2020-01-30
Building Real-Time ML Pipelines with Feature Stores and MLOps Frameworks
@ODSCAI
5.1K views•2022-02-20
Bypassing Tor Censorship: Bridges and Pluggable Transport Guide
@Coding_ForEveryone
397 views•2024-06-11
Neural Networks Explained: Math, Layers, and Learning Fundamentals
@3blue1brown
21.9M views•2017-10-05
Related Study Plans & Knowledge Roadmaps
Structured learning paths in Artificial Intelligence