How Outliers Impact Machine Learning Models | CPSC 330

Added:

The Informational Value of Anomalies and the Danger of Outlier Suppression

While robust regression techniques like Huber loss prevent extreme data points from distorting general models, a vital counter-perspective argues that outliers often contain the most critical information in a dataset. In domains such as fraud detection, network security, rare disease diagnosis, and financial risk modeling, the 'outliers' are not noise to be suppressed, but the primary phenomena of interest. Treating extreme values as nuisances can obscure rare but highly consequential events. Furthermore, in fields governed by heavy-tailed distributions, extreme observations are statistically expected and essential for accurate risk assessment. Rather than using robust methods to minimize their influence, practitioners should often employ specialized frameworks like Extreme Value Theory (EVT) or anomaly detection algorithms that specifically isolate, prioritize, and model these exceptional data points.

Fundamental concepts of supervised learning, specifically the distinction between regression and classification tasks.
The mechanics of Linear Regression, including how ordinary least squares (OLS) and Mean Squared Error (MSE) loss functions work.
Basic statistics, particularly data distributions, variance, and how outliers are defined in a dataset.
An understanding of standard loss functions (like MSE vs. MAE) and how they mathematically penalize prediction errors.
Alternative robust estimation algorithms, such as RANSAC (Random Sample Consensus) and the Theil-Sen estimator.
Unsupervised outlier and anomaly detection techniques, including Isolation Forests and Local Outlier Factor (LOF).
Data preprocessing and feature engineering strategies for handling outliers, such as winsorization, clipping, and robust scaling.
The computational trade-offs and optimization challenges of using non-quadratic loss functions in large-scale machine learning.
461 views6likes1:06:27@mgelbartOriginal Release: 2020-11-30

Outliers are observations that differ significantly from the majority of data, and they can severely impact machine learning models, particularly linear regression, by distorting coefficient estimates and reducing model performance. Outliers in training data can cause models to learn incorrect relationships, while outliers in deployment data can lead to unreliable predictions. Robust methods like HuberRegressor and random forests are more resistant to outliers compared to standard linear regression. The key distinction is between outliers in features (X) versus targets (Y), and between outliers in training versus deployment data, as each requires different handling strategies.